[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:08 dist-test-slave-ml7t krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:08 dist-test-slave-ml7t krb5kdc[399](info): set up 2 sockets
Sep 27 15:02:08 dist-test-slave-ml7t krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 15:02:09.530409 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:09 dist-test-slave-ml7t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:09.656567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:09.768622 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:09.768885 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:09.768929 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:09.773062 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:09.773130 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:09.773154 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:09.773186 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:09.773229 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:09.778530 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:09.779594 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:09.780637 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:09.789896 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:09.789896 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:09.789945 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:09.789964 412 server_base.cc:1034] running on GCE node
I20260927 15:02:09.790815 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:09.791922 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:09.793147 412 hybrid_clock.cc:648] HybridClock initialized: now 1790521329793073 us; error 87 us; skew 500 ppm
Sep 27 15:02:09 dist-test-slave-ml7t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:09.796051 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:09.796965 412 webserver.cc:460] Webserver started at http://127.0.98.62:33915/ using document root <none> and password file <none>
I20260927 15:02:09.797613 412 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:09.797703 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:09.797955 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:09.799983 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "8e7a8f5d097243c98d99c0d81f225b5b"
format_stamp: "Formatted at 2026-09-27 15:02:09 on dist-test-slave-ml7t"
server_key: "bd55869bb531ceece926018ba290a777"
server_key_iv: "2ddabd31fdce929a944792c3fa54054b"
server_key_version: "encryptionkey@0"
I20260927 15:02:09.800596 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e7a8f5d097243c98d99c0d81f225b5b"
format_stamp: "Formatted at 2026-09-27 15:02:09 on dist-test-slave-ml7t"
server_key: "bd55869bb531ceece926018ba290a777"
server_key_iv: "2ddabd31fdce929a944792c3fa54054b"
server_key_version: "encryptionkey@0"
I20260927 15:02:09.804323 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 15:02:09.806766 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:09.807964 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:09.808135 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "8e7a8f5d097243c98d99c0d81f225b5b"
format_stamp: "Formatted at 2026-09-27 15:02:09 on dist-test-slave-ml7t"
server_key: "bd55869bb531ceece926018ba290a777"
server_key_iv: "2ddabd31fdce929a944792c3fa54054b"
server_key_version: "encryptionkey@0"
I20260927 15:02:09.808254 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:09.833386 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:09.836385 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:09.836609 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:09.843816 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32845
I20260927 15:02:09.843840 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32845 every 8 connection(s)
I20260927 15:02:09.844746 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:09.847527 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:09.853039 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b: Bootstrap starting.
I20260927 15:02:09.853307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 412
I20260927 15:02:09.853469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:09.855103 392 external_mini_cluster.cc:1428] Setting key 977facb19f1be4c6c30c2ba188ba8d5d
I20260927 15:02:09.856228 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:09.857125 480 log.cc:824] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:09.859512 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b: No bootstrap required, opened a new log
I20260927 15:02:09.864645 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32845 } }
I20260927 15:02:09.864934 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:09.864988 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7a8f5d097243c98d99c0d81f225b5b, State: Initialized, Role: FOLLOWER
I20260927 15:02:09.865512 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32845 } }
I20260927 15:02:09.865679 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:09.865751 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:09.865906 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:09.867200 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32845 } }
I20260927 15:02:09.867575 480 leader_election.cc:302] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e7a8f5d097243c98d99c0d81f225b5b; no voters:
I20260927 15:02:09.867940 480 leader_election.cc:288] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:09.868064 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:09.868288 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [term 1 LEADER]: Becoming Leader. State: Replica: 8e7a8f5d097243c98d99c0d81f225b5b, State: Running, Role: LEADER
I20260927 15:02:09.868624 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32845 } }
I20260927 15:02:09.869266 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:09.870328 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32845 } } }
I20260927 15:02:09.870461 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:09.870747 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e7a8f5d097243c98d99c0d81f225b5b. Latest consensus state: current_term: 1 leader_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7a8f5d097243c98d99c0d81f225b5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32845 } } }
I20260927 15:02:09.870841 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:09.871078 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:09.874209 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 15:02:09 dist-test-slave-ml7t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:09.879573 494 catalog_manager.cc:1347] Generated new cluster ID: e0d451a6ac7e484da68bfe39096bf36a
I20260927 15:02:09.879632 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:09.885193 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:09.858596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.0.98.62:32845)
0927 15:02:09.859064 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:09.859135 (+ 71us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:09.860314 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 15:02:09.864256 (+ 3942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:09.864276 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:09.864603 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:09.864958 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:09.868017 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:09.869565 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:09.871473 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:09.872128 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:09.872301 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:09.877421 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:09.877428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:09.877443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:09.877475 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:09.879696 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:09.882278 (+ 2582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:09.882284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:09.882296 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:09.882411 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:09.882766 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:09.882769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:09.882771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:09.883165 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:09.883386 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:09.883897 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 15:02:09.884159 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":122,"threads_started":1}
I20260927 15:02:09.897516 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:09.898551 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:09.910815 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e7a8f5d097243c98d99c0d81f225b5b: Generated new TSK 0
I20260927 15:02:09.911633 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:09.948881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32845
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:10.079741 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:10.080106 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:10.080222 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:10.084240 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:10.084322 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:10.084422 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:10.090569 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:10.091601 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:10.092607 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:10.102074 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:10.102068 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:10.102324 506 server_base.cc:1034] running on GCE node
W20260927 15:02:10.102181 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:10.102807 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:10.103463 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:10.104694 506 hybrid_clock.cc:648] HybridClock initialized: now 1790521330104684 us; error 29 us; skew 500 ppm
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:10.107496 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:10.108321 506 webserver.cc:460] Webserver started at http://127.0.98.1:33631/ using document root <none> and password file <none>
I20260927 15:02:10.108896 506 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:10.108984 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:10.109208 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:10.111367 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "9503546df4094393ac98b380056faf38"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "069f2cc67d921283d424efa69e66f0b1"
server_key_iv: "c47245df4d1a2effd964e4ce52b694d7"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.111965 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9503546df4094393ac98b380056faf38"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "069f2cc67d921283d424efa69e66f0b1"
server_key_iv: "c47245df4d1a2effd964e4ce52b694d7"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.115726 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:02:10.118393 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.119652 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:10.119784 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "9503546df4094393ac98b380056faf38"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "069f2cc67d921283d424efa69e66f0b1"
server_key_iv: "c47245df4d1a2effd964e4ce52b694d7"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.119899 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:10.152594 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:10.155903 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:10.156101 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:10.156725 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:10.157570 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:10.157640 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.157701 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:10.157754 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.167520 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38493
I20260927 15:02:10.167549 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38493 every 8 connection(s)
I20260927 15:02:10.168388 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:10.176347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 506
I20260927 15:02:10.176493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:10.176784 392 external_mini_cluster.cc:1428] Setting key 2cb506ec57b838a9fe0ec58cb44cda9b
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:10.183434 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.170300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41285 (local address 127.0.98.62:32845)
0927 15:02:10.170485 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:10.170491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:10.171402 (+ 911us) server_negotiation.cc:407] Connection header received
0927 15:02:10.172727 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:10.172731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:10.172792 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:10.172872 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:10.174041 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.174581 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.175741 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.175980 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.176118 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.179015 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:10.179018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:10.179020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:10.179044 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:10.181018 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.181581 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.181584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.181586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.181694 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.182180 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.182183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.182184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.182375 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:10.182453 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:10.183182 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 15:02:10.183308 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:02:10.184772 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.170802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32845 (local address 127.0.98.1:41285)
0927 15:02:10.171268 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:10.171310 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:10.172522 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:10.173030 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:10.173037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:10.173381 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:10.173899 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:10.173914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.174714 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.174718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:10.175595 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:10.175604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.176152 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.176157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:10.176329 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.176958 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:10.176974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:10.178831 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:10.181173 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:10.181178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:10.181192 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:10.181475 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:10.181840 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:10.181847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:10.181852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:10.182068 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:10.182489 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:10.182498 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:10.182868 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 15:02:10.183115 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 15:02:10.183555 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260927 15:02:10.186134 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32845
I20260927 15:02:10.186400 635 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:10.186919 635 heartbeater.cc:511] Master 127.0.98.62:32845 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:02:10.188416 444 ts_manager.cc:194] Registered new tserver with Master: 9503546df4094393ac98b380056faf38 (127.0.98.1:38493)
I20260927 15:02:10.189659 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:10.204914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32845
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:10.313736 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:10.314020 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:10.314066 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:10.318158 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:10.318222 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:10.318284 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:10.323570 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:10.324615 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:10.325559 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:10.335546 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:10.335546 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:10.335635 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:10.335729 640 server_base.cc:1034] running on GCE node
I20260927 15:02:10.336284 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:10.336953 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:10.338146 640 hybrid_clock.cc:648] HybridClock initialized: now 1790521330338128 us; error 28 us; skew 500 ppm
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:10.341465 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:10.342419 640 webserver.cc:460] Webserver started at http://127.0.98.2:40361/ using document root <none> and password file <none>
I20260927 15:02:10.342990 640 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:10.343128 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:10.343354 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:10.345351 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab574460a93e42f8bdfbc63b7c129820"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "4628f3a0cf7ef9d54f0225a683a36cac"
server_key_iv: "94c4c0e7610eccd2ab19ab3531a8ffa8"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.345961 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab574460a93e42f8bdfbc63b7c129820"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "4628f3a0cf7ef9d54f0225a683a36cac"
server_key_iv: "94c4c0e7610eccd2ab19ab3531a8ffa8"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.349476 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:10.351889 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.353060 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:10.353204 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "ab574460a93e42f8bdfbc63b7c129820"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "4628f3a0cf7ef9d54f0225a683a36cac"
server_key_iv: "94c4c0e7610eccd2ab19ab3531a8ffa8"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.353310 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:10.380079 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:10.383051 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:10.383285 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:10.383853 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:10.384655 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:10.384723 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.384795 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:10.384848 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.394711 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41409
I20260927 15:02:10.394730 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41409 every 8 connection(s)
I20260927 15:02:10.395571 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:10.401494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 640
I20260927 15:02:10.401623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:10.401906 392 external_mini_cluster.cc:1428] Setting key 6c02d98ae554d3ff65280f8ca9894686
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:10.410079 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.397312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45689 (local address 127.0.98.62:32845)
0927 15:02:10.397455 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:10.397460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:10.398384 (+ 924us) server_negotiation.cc:407] Connection header received
0927 15:02:10.399672 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:10.399675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:10.399720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:10.399796 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:10.400900 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.401469 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.402963 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.403312 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.403408 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.406584 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:10.406587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:10.406589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:10.406611 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:10.408039 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.408555 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.408559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.408561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.408638 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.408970 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.408973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.408975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.409153 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:10.409258 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:10.409734 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 15:02:10.409878 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:10.410965 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.397772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32845 (local address 127.0.98.2:45689)
0927 15:02:10.398237 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:10.398277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:10.399470 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:10.399955 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:10.399962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:10.400330 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:10.400777 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:10.400789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.401615 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.401620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:10.402849 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:10.402858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.403474 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.403478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:10.403692 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.404319 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:10.404345 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:10.406417 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:10.408178 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:10.408183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:10.408197 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:10.408449 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:10.408734 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:10.408737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:10.408739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:10.408872 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:10.409254 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:10.409260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:10.409493 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 15:02:10.409681 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 15:02:10.410005 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260927 15:02:10.412179 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32845
I20260927 15:02:10.412412 769 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:10.412921 769 heartbeater.cc:511] Master 127.0.98.62:32845 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:10.414003 444 ts_manager.cc:194] Registered new tserver with Master: ab574460a93e42f8bdfbc63b7c129820 (127.0.98.2:41409)
I20260927 15:02:10.414561 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:10.429342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32845
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:10.543197 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:10.543442 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:10.543485 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:10.547562 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:10.547626 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:10.547688 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:10.552760 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:10.553709 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:10.554767 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:10.564574 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:10.564666 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:10.564682 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:10.564891 774 server_base.cc:1034] running on GCE node
I20260927 15:02:10.565366 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:10.566049 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:10.567235 774 hybrid_clock.cc:648] HybridClock initialized: now 1790521330567219 us; error 28 us; skew 500 ppm
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:10.570302 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:10.571259 774 webserver.cc:460] Webserver started at http://127.0.98.3:41015/ using document root <none> and password file <none>
I20260927 15:02:10.571831 774 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:10.571921 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:10.572148 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:10.574275 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "e99a0b957dd8491392190b43fb836382"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "27e740627a38543450002cf021197747"
server_key_iv: "d523a6d07a0ffa0593f6af3b7dc8a847"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.574877 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e99a0b957dd8491392190b43fb836382"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "27e740627a38543450002cf021197747"
server_key_iv: "d523a6d07a0ffa0593f6af3b7dc8a847"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.578600 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:10.580915 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.582206 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:10.582336 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "e99a0b957dd8491392190b43fb836382"
format_stamp: "Formatted at 2026-09-27 15:02:10 on dist-test-slave-ml7t"
server_key: "27e740627a38543450002cf021197747"
server_key_iv: "d523a6d07a0ffa0593f6af3b7dc8a847"
server_key_version: "encryptionkey@0"
I20260927 15:02:10.582443 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:10.616097 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:10.619648 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:10.619854 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:10.620442 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:10.621232 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:10.621276 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.621343 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:10.621385 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:10.630697 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44019
I20260927 15:02:10.630762 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44019 every 8 connection(s)
I20260927 15:02:10.631595 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:10.635869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 774
I20260927 15:02:10.635998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:10.636274 392 external_mini_cluster.cc:1428] Setting key 0dcd6a4850127e1e7a2a06da0b335d6d
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:10.646100 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.633405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54959 (local address 127.0.98.62:32845)
0927 15:02:10.633582 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:10.633586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:10.634824 (+ 1238us) server_negotiation.cc:407] Connection header received
0927 15:02:10.636282 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:10.636285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:10.636341 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:10.636431 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:10.638043 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.638727 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.639848 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.640066 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.640181 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.642606 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:10.642609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:10.642611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:10.642636 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:10.644262 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.644715 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.644718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.644719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.644797 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.645121 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.645123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.645124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.645296 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:10.645375 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:10.645881 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 15:02:10.645975 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:02:10.646837 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.633817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32845 (local address 127.0.98.3:54959)
0927 15:02:10.634281 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:10.634324 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:10.636012 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:10.636653 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:10.636662 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:10.637141 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:10.637892 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:10.637912 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.638854 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.638857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:10.639729 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:10.639737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.640214 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.640218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:10.640389 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.640904 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:10.640919 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:10.642419 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:10.644367 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:10.644371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:10.644382 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:10.644611 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:10.644873 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:10.644875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:10.644876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:10.644990 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:10.645386 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:10.645391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:10.645597 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 15:02:10.645755 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:02:10.646048 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":109,"threads_started":1}
I20260927 15:02:10.648012 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32845
I20260927 15:02:10.648227 903 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:10.648629 903 heartbeater.cc:511] Master 127.0.98.62:32845 requested a full tablet report, sending...
I20260927 15:02:10.649646 444 ts_manager.cc:194] Registered new tserver with Master: e99a0b957dd8491392190b43fb836382 (127.0.98.3:44019)
I20260927 15:02:10.650463 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54959
I20260927 15:02:10.650874 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:10.676800 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.667173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.0.98.1:38493)
0927 15:02:10.667508 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:10.667523 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:10.667547 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:02:10.667662 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:10.667670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:10.667870 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:10.668033 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:10.668722 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.669555 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.670581 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.670948 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.671071 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.673437 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:10.673445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:10.673459 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:10.673481 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:10.675034 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.675477 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.675482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.675484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.675559 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.675837 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.675841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.675844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.676080 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:10.676294 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:10.676466 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 15:02:10.676619 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":104,"threads_started":1}
W20260927 15:02:10.677750 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58594
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:10.703402 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:10.695131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.0.98.1:38493)
0927 15:02:10.695289 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:10.695294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:10.695396 (+ 102us) server_negotiation.cc:407] Connection header received
0927 15:02:10.695561 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:10.695564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:10.695615 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:10.695719 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:10.696392 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.696911 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.697750 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:10.698024 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:10.698139 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:10.700467 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:10.700470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:10.700472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:10.700495 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:10.701729 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.702209 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.702214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.702216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.702297 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:10.702571 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:10.702574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:10.702576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:10.702746 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:10.702882 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:10.703118 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 15:02:10.703233 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:02:10.704387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 506
I20260927 15:02:10.710153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 640
I20260927 15:02:10.717643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 774
I20260927 15:02:10.722792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 412
2026-09-27T15:02:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2261 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[917](info): set up 2 sockets
Sep 27 15:02:10 dist-test-slave-ml7t krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 15:02:11.789361 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:11 dist-test-slave-ml7t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:11.912740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:12.019304 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:12.019567 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:12.019652 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:12.023792 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:12.023854 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:12.023874 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:12.023892 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:12.023907 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:12.029261 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:12.030428 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:12.031459 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:12.039793 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:12.039945 930 server_base.cc:1034] running on GCE node
W20260927 15:02:12.039773 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:12.039773 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:12.040668 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:12.041747 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:12.042986 930 hybrid_clock.cc:648] HybridClock initialized: now 1790521332042971 us; error 33 us; skew 500 ppm
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:12.045706 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:12.046628 930 webserver.cc:460] Webserver started at http://127.0.98.62:43513/ using document root <none> and password file <none>
I20260927 15:02:12.047291 930 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:12.047398 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:12.047652 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:12.049674 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "222504f1d3c24ae0a0a547f64f5733f1"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "1c9b67cedf2d4281c4be976347fd45c7"
server_key_iv: "acd728986f9c81494e925ea9bf25907c"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.050295 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "222504f1d3c24ae0a0a547f64f5733f1"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "1c9b67cedf2d4281c4be976347fd45c7"
server_key_iv: "acd728986f9c81494e925ea9bf25907c"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.053927 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:12.056422 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.057632 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:12.057819 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "222504f1d3c24ae0a0a547f64f5733f1"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "1c9b67cedf2d4281c4be976347fd45c7"
server_key_iv: "acd728986f9c81494e925ea9bf25907c"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.057936 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:12.074965 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:12.078372 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:12.078575 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:12.085304 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35371
I20260927 15:02:12.085330 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35371 every 8 connection(s)
I20260927 15:02:12.086249 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:12.088763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 930
I20260927 15:02:12.088856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:12.088941 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:12.089176 392 external_mini_cluster.cc:1428] Setting key 36b14de4f50768abee94bd496dd76fed
I20260927 15:02:12.096027 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1: Bootstrap starting.
I20260927 15:02:12.098474 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1: Neither blocks nor log segments found. Creating new log.
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:12.099375 998 log.cc:824] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:12.101406 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1: No bootstrap required, opened a new log
I20260927 15:02:12.104089 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.090624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.0.98.62:35371)
0927 15:02:12.091042 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.091059 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.091112 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:02:12.092319 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.092344 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.092766 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.093150 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:12.093830 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.095337 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.096184 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.096654 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.096847 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.099660 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:12.099666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:12.099678 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:12.099713 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:12.101426 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.101926 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.101934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.101944 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.102051 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.102344 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.102347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.102348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.102708 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:12.102910 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.103147 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.103404 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
I20260927 15:02:12.104966 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222504f1d3c24ae0a0a547f64f5733f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35371 } }
I20260927 15:02:12.105168 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:12.105242 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222504f1d3c24ae0a0a547f64f5733f1, State: Initialized, Role: FOLLOWER
I20260927 15:02:12.105680 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222504f1d3c24ae0a0a547f64f5733f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35371 } }
I20260927 15:02:12.105887 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:12.105952 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:12.106052 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:12.107067 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222504f1d3c24ae0a0a547f64f5733f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35371 } }
I20260927 15:02:12.107347 998 leader_election.cc:302] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 222504f1d3c24ae0a0a547f64f5733f1; no voters:
I20260927 15:02:12.107589 998 leader_election.cc:288] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:12.107777 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:12.107998 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [term 1 LEADER]: Becoming Leader. State: Replica: 222504f1d3c24ae0a0a547f64f5733f1, State: Running, Role: LEADER
I20260927 15:02:12.108310 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222504f1d3c24ae0a0a547f64f5733f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35371 } }
I20260927 15:02:12.108817 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:12.109462 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222504f1d3c24ae0a0a547f64f5733f1. Latest consensus state: current_term: 1 leader_uuid: "222504f1d3c24ae0a0a547f64f5733f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222504f1d3c24ae0a0a547f64f5733f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35371 } } }
I20260927 15:02:12.109612 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:12.109999 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "222504f1d3c24ae0a0a547f64f5733f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222504f1d3c24ae0a0a547f64f5733f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35371 } } }
I20260927 15:02:12.110087 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:12.110103 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:12.112998 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:12.119346 1012 catalog_manager.cc:1347] Generated new cluster ID: 083796c44c5e4cca8917382882f642c1
I20260927 15:02:12.119431 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:12.138960 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:12.140012 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:12.147174 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 222504f1d3c24ae0a0a547f64f5733f1: Generated new TSK 0
I20260927 15:02:12.147897 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:12.181493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35371
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:12.295624 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:12.295960 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:12.296058 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:12.300179 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:12.300279 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:12.300380 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:12.305662 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:12.306813 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:12.307917 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:12.317850 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:12.317902 1024 server_base.cc:1034] running on GCE node
W20260927 15:02:12.317850 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:12.317881 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:12.318434 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:12.319111 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:12.320302 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790521332320288 us; error 26 us; skew 500 ppm
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:12.323174 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:12.324074 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34471/ using document root <none> and password file <none>
I20260927 15:02:12.324642 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:12.324738 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:12.324980 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:12.327068 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "bc0ae221fb7c41fb87129b0b7d194d30"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "bd15f18283ac27f6f09ec06c06c79ac7"
server_key_iv: "c29e16a6ab1c7b04294173d53d6183e7"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.327689 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bc0ae221fb7c41fb87129b0b7d194d30"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "bd15f18283ac27f6f09ec06c06c79ac7"
server_key_iv: "c29e16a6ab1c7b04294173d53d6183e7"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.331575 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:02:12.334053 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.335311 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:12.335457 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "bc0ae221fb7c41fb87129b0b7d194d30"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "bd15f18283ac27f6f09ec06c06c79ac7"
server_key_iv: "c29e16a6ab1c7b04294173d53d6183e7"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.335595 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:12.360473 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:12.363633 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:12.363826 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:12.364403 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:12.365211 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:12.365280 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.365350 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:12.365402 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.374467 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38727
I20260927 15:02:12.374500 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38727 every 8 connection(s)
I20260927 15:02:12.375308 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:12.378158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 1024
I20260927 15:02:12.378265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:12.378571 392 external_mini_cluster.cc:1428] Setting key 973fdba8a9860ddcdab4ea462cedb0ed
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:12.391233 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.376969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33343 (local address 127.0.98.62:35371)
0927 15:02:12.377183 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.377188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.378052 (+ 864us) server_negotiation.cc:407] Connection header received
0927 15:02:12.379646 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.379649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.379692 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.379784 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:12.381466 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.382193 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.384081 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.384315 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.384448 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.387539 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:12.387542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:12.387550 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:12.387572 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:12.389035 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.389584 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.389587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.389588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.389663 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.390034 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.390038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.390039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.390228 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:12.390308 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.390849 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.390999 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 15:02:12.392135 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.377397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35371 (local address 127.0.98.1:33343)
0927 15:02:12.377875 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:12.377941 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:12.379451 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:12.379961 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:12.379977 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:12.380524 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:12.381227 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.381249 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.382334 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.382339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.383878 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.383893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.384484 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.384489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.384744 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.385619 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:12.385635 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:12.387353 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:12.389152 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:12.389157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:12.389173 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:12.389483 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:12.389760 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:12.389762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:12.389764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:12.389944 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:12.390330 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:12.390336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:12.390566 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 15:02:12.390741 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:02:12.391089 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":120,"threads_started":1}
I20260927 15:02:12.393501 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35371
I20260927 15:02:12.393749 1153 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:12.394243 1153 heartbeater.cc:511] Master 127.0.98.62:35371 requested a full tablet report, sending...
I20260927 15:02:12.395670 962 ts_manager.cc:194] Registered new tserver with Master: bc0ae221fb7c41fb87129b0b7d194d30 (127.0.98.1:38727)
I20260927 15:02:12.396826 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:12.409237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35371
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:12.523821 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:12.524050 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:12.524092 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:12.528041 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:12.528093 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:12.528149 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:12.533136 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:12.534108 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:12.535023 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:12.544104 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:12.544222 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:12.544222 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:12.544435 1158 server_base.cc:1034] running on GCE node
I20260927 15:02:12.544867 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:12.545509 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:12.546718 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790521332546677 us; error 56 us; skew 500 ppm
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:12.549297 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:12.550226 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43237/ using document root <none> and password file <none>
I20260927 15:02:12.550761 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:12.550817 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:12.550969 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:12.552875 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "d19637b338594fd59495d5e6b1473a44"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "576dabcb8a63b75961f6d1b26f6e95e9"
server_key_iv: "2db1438e9007bf4c863e27ecf45ae013"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.553344 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d19637b338594fd59495d5e6b1473a44"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "576dabcb8a63b75961f6d1b26f6e95e9"
server_key_iv: "2db1438e9007bf4c863e27ecf45ae013"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.556823 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:02:12.559201 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.560410 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:12.560499 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "d19637b338594fd59495d5e6b1473a44"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "576dabcb8a63b75961f6d1b26f6e95e9"
server_key_iv: "2db1438e9007bf4c863e27ecf45ae013"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.560624 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:12.570887 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:12.573576 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:12.573822 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:12.574396 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:12.575120 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:12.575165 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.575202 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:12.575217 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.584276 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35081
I20260927 15:02:12.584311 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35081 every 8 connection(s)
I20260927 15:02:12.585143 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:12.589085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 1158
I20260927 15:02:12.589217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:12.589464 392 external_mini_cluster.cc:1428] Setting key 7d4781e1a0499d734bdcfb984544bfc3
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:12.599768 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.587015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38351 (local address 127.0.98.62:35371)
0927 15:02:12.587156 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.587160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.588317 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 15:02:12.589263 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.589266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.589324 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.589440 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:12.590933 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.591712 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.592946 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.593173 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.593287 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.595964 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:12.595971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:12.595976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:12.596011 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:12.597650 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.598174 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.598181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.598185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.598268 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.598610 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.598615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.598617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.598863 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:12.598977 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.599489 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.599639 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:02:12.600543 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.587259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35371 (local address 127.0.98.2:38351)
0927 15:02:12.588157 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:12.588199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:12.589050 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:12.589614 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:12.589621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:12.590130 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:12.590790 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.590811 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.591850 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.591853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.592815 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.592824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.593320 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.593325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.593489 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.594086 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:12.594109 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:12.595798 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:12.597812 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:12.597822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:12.597833 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:12.598075 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:12.598371 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:12.598373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:12.598374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:12.598495 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:12.598973 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:12.598977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:12.599210 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 15:02:12.599398 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 15:02:12.599656 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":528,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:12.601761 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35371
I20260927 15:02:12.602061 1287 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:12.602557 1287 heartbeater.cc:511] Master 127.0.98.62:35371 requested a full tablet report, sending...
I20260927 15:02:12.603602 962 ts_manager.cc:194] Registered new tserver with Master: d19637b338594fd59495d5e6b1473a44 (127.0.98.2:35081)
I20260927 15:02:12.604154 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:12.617156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35371
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:12.727516 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:12.727807 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:12.727882 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:12.731918 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:12.732007 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:12.732110 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:12.737352 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:12.738454 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:12.739547 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:12.749531 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:12.749511 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:12.749511 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:12.749887 1292 server_base.cc:1034] running on GCE node
I20260927 15:02:12.750425 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:12.751084 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:12.752279 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790521332752261 us; error 31 us; skew 500 ppm
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:12.754966 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:12.755829 1292 webserver.cc:460] Webserver started at http://127.0.98.3:45337/ using document root <none> and password file <none>
I20260927 15:02:12.756392 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:12.756484 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:12.756717 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:12.758795 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "217a15d7fb5f4ec883dd7d756a76b52f"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "ca243cc3752ab23eb751058699c3a949"
server_key_iv: "f5e2e784290da6eb5f806e8f46220d54"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.759336 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "217a15d7fb5f4ec883dd7d756a76b52f"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "ca243cc3752ab23eb751058699c3a949"
server_key_iv: "f5e2e784290da6eb5f806e8f46220d54"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.763013 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:12.765698 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.767025 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:12.767163 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "217a15d7fb5f4ec883dd7d756a76b52f"
format_stamp: "Formatted at 2026-09-27 15:02:12 on dist-test-slave-ml7t"
server_key: "ca243cc3752ab23eb751058699c3a949"
server_key_iv: "f5e2e784290da6eb5f806e8f46220d54"
server_key_version: "encryptionkey@0"
I20260927 15:02:12.767277 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:12.779320 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:12.782171 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:12.782361 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:12.782989 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:12.783788 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:12.783876 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.783947 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:12.783982 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:12.793076 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39059
I20260927 15:02:12.793108 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39059 every 8 connection(s)
I20260927 15:02:12.794003 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:12.803189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 1292
I20260927 15:02:12.803289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
Sep 27 15:02:12 dist-test-slave-ml7t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:12.803565 392 external_mini_cluster.cc:1428] Setting key e00e16e95f0098149d7b2facb3e98363
I20260927 15:02:12.807933 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.795801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55253 (local address 127.0.98.62:35371)
0927 15:02:12.795950 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.795954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.796825 (+ 871us) server_negotiation.cc:407] Connection header received
0927 15:02:12.798191 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.798193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.798239 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.798313 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:12.799472 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.800044 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.801244 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.801454 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.801556 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.804224 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:12.804232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:12.804234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:12.804258 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:12.805927 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.806413 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.806418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.806420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.806510 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.806848 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.806851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.806853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.807039 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:12.807144 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.807624 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.807755 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:02:12.808825 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.796263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35371 (local address 127.0.98.3:55253)
0927 15:02:12.796676 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:12.796718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:12.797953 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:12.798480 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:12.798486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:12.798821 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:12.799317 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.799330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.800169 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.800171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.801022 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.801030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.801606 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.801609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.801742 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.802413 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:12.802429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:12.804044 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:12.806038 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:12.806043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:12.806058 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:12.806309 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:12.806602 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:12.806606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:12.806609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:12.806748 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:12.807147 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:12.807156 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:12.807397 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 15:02:12.807585 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 15:02:12.807953 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260927 15:02:12.810050 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35371
I20260927 15:02:12.810303 1421 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:12.810792 1421 heartbeater.cc:511] Master 127.0.98.62:35371 requested a full tablet report, sending...
I20260927 15:02:12.811825 962 ts_manager.cc:194] Registered new tserver with Master: 217a15d7fb5f4ec883dd7d756a76b52f (127.0.98.3:39059)
I20260927 15:02:12.812413 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55253
I20260927 15:02:12.817497 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:02:12.828555 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.820885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.0.98.62:35371)
0927 15:02:12.821148 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.821153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.821302 (+ 149us) server_negotiation.cc:407] Connection header received
0927 15:02:12.821457 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.821460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.821510 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.821595 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:12.822416 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.822928 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.823764 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.823976 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.824096 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.825460 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:12.825464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:12.825466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:12.825498 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:12.826899 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.827320 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.827325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.827327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.827401 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:12.827685 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:12.827688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:12.827689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:12.827899 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:12.828048 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.828275 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.828387 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 15:02:12.833634 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:12.836743 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:12.851419 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.845684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.0.98.1:38727)
0927 15:02:12.846522 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.846540 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.846564 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:02:12.846641 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.846648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.846857 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.847416 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:12.847621 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.848767 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.850169 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.850696 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.850817 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.850961 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.851156 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.851249 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":106,"threads_started":1}
I20260927 15:02:12.851414 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.845374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38727 (local address 127.0.0.1:55756)
0927 15:02:12.845725 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:12.845802 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:12.845923 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:12.847050 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:12.847056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:12.847082 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:12.847260 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.847267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.848909 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.848912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.850056 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.850063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.850816 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.850819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.850919 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.851004 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 15:02:12.851129 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:02:12.851234 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":108,"thread_start_us":87,"threads_started":1}
I20260927 15:02:12.851421 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.846157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35081 (local address 127.0.0.1:49620)
0927 15:02:12.846360 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:12.846374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:12.846458 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:12.847263 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:12.847266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:12.847282 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:12.847381 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.847387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.848423 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.848428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.849715 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.849725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.850859 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.850862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.850981 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.851009 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:02:12.851093 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:02:12.851234 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260927 15:02:12.851955 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.846452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.98.2:35081)
0927 15:02:12.846797 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.846816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.846840 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:02:12.846912 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.846917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.847117 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.847295 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:12.847466 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.848295 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.849895 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.850721 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.851502 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.851682 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.851729 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.851811 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":157,"threads_started":1}
I20260927 15:02:12.852285 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.845405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39059 (local address 127.0.0.1:59918)
0927 15:02:12.846192 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:12.846207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:12.846306 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:12.847462 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:12.847465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:12.847482 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:12.847574 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.847579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.848943 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.848946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.850182 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:12.850192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.851420 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.851424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:12.851985 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.852014 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:02:12.852091 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:02:12.852163 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":94,"thread_start_us":133,"threads_started":1}
I20260927 15:02:12.852445 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:12.846159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.0.98.3:39059)
0927 15:02:12.846914 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:12.846925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:12.846944 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:02:12.847005 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:12.847009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:12.847166 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:12.847294 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:12.847677 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.848788 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.850330 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:12.851134 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:12.851228 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:12.851362 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:12.852159 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 15:02:12.852249 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":95,"threads_started":1}
I20260927 15:02:12.854811 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 59cceaaea8eb48669794198ed5f73610 (DEFAULT_TABLE table=test-table [id=6084b2e51fe948f0a2a9c79ae468b729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:12.854887 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 59cceaaea8eb48669794198ed5f73610 (DEFAULT_TABLE table=test-table [id=6084b2e51fe948f0a2a9c79ae468b729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:12.854890 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 59cceaaea8eb48669794198ed5f73610 (DEFAULT_TABLE table=test-table [id=6084b2e51fe948f0a2a9c79ae468b729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:12.855815 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cceaaea8eb48669794198ed5f73610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:12.855883 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cceaaea8eb48669794198ed5f73610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:12.855882 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cceaaea8eb48669794198ed5f73610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:12.862013 1438 tablet_bootstrap.cc:492] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44: Bootstrap starting.
I20260927 15:02:12.863296 1439 tablet_bootstrap.cc:492] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f: Bootstrap starting.
I20260927 15:02:12.863744 1440 tablet_bootstrap.cc:492] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30: Bootstrap starting.
I20260927 15:02:12.864779 1438 tablet_bootstrap.cc:654] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:12.865587 1440 tablet_bootstrap.cc:654] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:12.865900 1438 log.cc:824] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:12.866910 1440 log.cc:824] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:12.867780 1439 tablet_bootstrap.cc:654] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:12.868048 1438 tablet_bootstrap.cc:492] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44: No bootstrap required, opened a new log
I20260927 15:02:12.868254 1438 ts_tablet_manager.cc:1397] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:02:12.868919 1440 tablet_bootstrap.cc:492] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30: No bootstrap required, opened a new log
I20260927 15:02:12.868981 1439 log.cc:824] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:12.869118 1440 ts_tablet_manager.cc:1397] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:02:12.871400 1439 tablet_bootstrap.cc:492] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f: No bootstrap required, opened a new log
I20260927 15:02:12.871618 1439 ts_tablet_manager.cc:1397] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 15:02:12.872249 1438 raft_consensus.cc:348] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:12.872493 1438 raft_consensus.cc:374] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:12.872545 1438 raft_consensus.cc:729] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19637b338594fd59495d5e6b1473a44, State: Initialized, Role: FOLLOWER
I20260927 15:02:12.872977 1438 consensus_queue.cc:260] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:12.872854 1440 raft_consensus.cc:348] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:12.873145 1440 raft_consensus.cc:374] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:12.873211 1440 raft_consensus.cc:729] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0ae221fb7c41fb87129b0b7d194d30, State: Initialized, Role: FOLLOWER
I20260927 15:02:12.873656 1440 consensus_queue.cc:260] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:12.874018 1438 ts_tablet_manager.cc:1428] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:02:12.874369 1287 heartbeater.cc:503] Master 127.0.98.62:35371 was elected leader, sending a full tablet report...
I20260927 15:02:12.874631 1440 ts_tablet_manager.cc:1428] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 15:02:12.874965 1153 heartbeater.cc:503] Master 127.0.98.62:35371 was elected leader, sending a full tablet report...
I20260927 15:02:12.875355 1439 raft_consensus.cc:348] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:12.875599 1439 raft_consensus.cc:374] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:12.875669 1439 raft_consensus.cc:729] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217a15d7fb5f4ec883dd7d756a76b52f, State: Initialized, Role: FOLLOWER
I20260927 15:02:12.876078 1439 consensus_queue.cc:260] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:12.876753 1421 heartbeater.cc:503] Master 127.0.98.62:35371 was elected leader, sending a full tablet report...
I20260927 15:02:12.876976 1439 ts_tablet_manager.cc:1428] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 15:02:12.877432 1154 tablet.cc:2369] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:13.044942 1422 tablet.cc:2369] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:13.086272 1288 tablet.cc:2369] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:13.112640 1446 raft_consensus.cc:482] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:13.112848 1446 raft_consensus.cc:504] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:13.114130 1446 leader_election.cc:288] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc0ae221fb7c41fb87129b0b7d194d30 (127.0.98.1:38727), d19637b338594fd59495d5e6b1473a44 (127.0.98.2:35081)
I20260927 15:02:13.118250 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:13.114474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38483 (local address 127.0.98.1:38727)
0927 15:02:13.114614 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:13.114619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:13.114634 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:02:13.114775 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:13.114778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:13.114834 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:13.114904 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:13.115306 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.115942 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.117131 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.117650 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.117726 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:13.117824 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:13.118047 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 15:02:13.118131 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:02:13.118254 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:13.114347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38727 (local address 127.0.98.3:38483)
0927 15:02:13.114486 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:13.114500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:13.114638 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:13.115022 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:13.115024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:13.115040 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:13.115166 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:13.115175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.116042 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.116045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:13.117015 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:13.117024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.117750 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.117754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:13.117913 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:13.117938 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:02:13.118021 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:02:13.118100 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":37}
I20260927 15:02:13.118597 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:13.114771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43445 (local address 127.0.98.2:35081)
0927 15:02:13.114918 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:13.114922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:13.115182 (+ 260us) server_negotiation.cc:407] Connection header received
0927 15:02:13.115288 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:13.115291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:13.115344 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:13.115403 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:13.115670 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.116248 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.117488 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.118046 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.118148 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:13.118203 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:13.118408 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 15:02:13.118487 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:02:13.118930 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cceaaea8eb48669794198ed5f73610" candidate_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" is_pre_election: true
I20260927 15:02:13.119030 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:13.114663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35081 (local address 127.0.98.3:43445)
0927 15:02:13.115088 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:13.115099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:13.115185 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:13.115432 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:13.115435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:13.115451 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:13.115570 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:13.115576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.116351 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.116353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:13.117347 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:13.117356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.118150 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.118154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:13.118247 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:13.118278 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:02:13.118771 (+ 493us) client_negotiation.cc:240] Negotiation successful
0927 15:02:13.118846 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":40,"thread_start_us":151,"threads_started":1}
I20260927 15:02:13.119234 1107 raft_consensus.cc:2454] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217a15d7fb5f4ec883dd7d756a76b52f in term 0.
I20260927 15:02:13.119711 1309 leader_election.cc:302] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a15d7fb5f4ec883dd7d756a76b52f, bc0ae221fb7c41fb87129b0b7d194d30; no voters:
I20260927 15:02:13.119978 1446 raft_consensus.cc:2781] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:13.119961 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cceaaea8eb48669794198ed5f73610" candidate_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19637b338594fd59495d5e6b1473a44" is_pre_election: true
I20260927 15:02:13.120088 1446 raft_consensus.cc:482] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:13.120149 1446 raft_consensus.cc:3037] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:13.120261 1241 raft_consensus.cc:2454] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217a15d7fb5f4ec883dd7d756a76b52f in term 0.
I20260927 15:02:13.121515 1446 raft_consensus.cc:504] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:13.121929 1446 leader_election.cc:288] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [CANDIDATE]: Term 1 election: Requested vote from peers bc0ae221fb7c41fb87129b0b7d194d30 (127.0.98.1:38727), d19637b338594fd59495d5e6b1473a44 (127.0.98.2:35081)
I20260927 15:02:13.122351 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cceaaea8eb48669794198ed5f73610" candidate_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc0ae221fb7c41fb87129b0b7d194d30"
I20260927 15:02:13.122424 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cceaaea8eb48669794198ed5f73610" candidate_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19637b338594fd59495d5e6b1473a44"
I20260927 15:02:13.122466 1107 raft_consensus.cc:3037] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:13.122538 1241 raft_consensus.cc:3037] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:13.123389 1107 raft_consensus.cc:2454] T 59cceaaea8eb48669794198ed5f73610 P bc0ae221fb7c41fb87129b0b7d194d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217a15d7fb5f4ec883dd7d756a76b52f in term 1.
I20260927 15:02:13.123557 1241 raft_consensus.cc:2454] T 59cceaaea8eb48669794198ed5f73610 P d19637b338594fd59495d5e6b1473a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217a15d7fb5f4ec883dd7d756a76b52f in term 1.
I20260927 15:02:13.123636 1309 leader_election.cc:302] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a15d7fb5f4ec883dd7d756a76b52f, bc0ae221fb7c41fb87129b0b7d194d30; no voters:
I20260927 15:02:13.123896 1446 raft_consensus.cc:2781] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:13.124114 1446 raft_consensus.cc:686] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [term 1 LEADER]: Becoming Leader. State: Replica: 217a15d7fb5f4ec883dd7d756a76b52f, State: Running, Role: LEADER
I20260927 15:02:13.124428 1446 consensus_queue.cc:237] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } }
I20260927 15:02:13.127609 961 catalog_manager.cc:5495] T 59cceaaea8eb48669794198ed5f73610 P 217a15d7fb5f4ec883dd7d756a76b52f reported cstate change: term changed from 0 to 1, leader changed from <none> to 217a15d7fb5f4ec883dd7d756a76b52f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc0ae221fb7c41fb87129b0b7d194d30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "217a15d7fb5f4ec883dd7d756a76b52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19637b338594fd59495d5e6b1473a44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35081 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:13.148729 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:13.139832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.0.98.1:38727)
0927 15:02:13.139981 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:13.139986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:13.140145 (+ 159us) server_negotiation.cc:407] Connection header received
0927 15:02:13.140347 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:13.140349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:13.140393 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:13.140478 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:13.141186 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.141687 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.142569 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.142795 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.142875 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:13.145326 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:13.145336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:13.145350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:13.145373 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:13.147087 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:13.147501 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:13.147507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:13.147508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:13.147603 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:13.147831 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:13.147833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:13.147834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:13.148067 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:13.148226 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:13.148401 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 15:02:13.148495 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:13.174852 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:13.166533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.0.98.1:38727)
0927 15:02:13.166709 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:13.166713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:13.166781 (+ 68us) server_negotiation.cc:407] Connection header received
0927 15:02:13.166893 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:13.166895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:13.166935 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:13.167007 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:13.167698 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.168199 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.169020 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.169228 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.169301 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:13.171897 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:13.171900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:13.171902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:13.171923 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:13.173222 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:13.173652 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:13.173654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:13.173655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:13.173724 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:13.174000 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:13.174002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:13.174003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:13.174153 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:13.174251 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:13.174554 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:02:13.174653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:13.199901 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:13.192168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.0.98.1:38727)
0927 15:02:13.192318 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:13.192322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:13.192374 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:02:13.192477 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:13.192480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:13.192521 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:13.192602 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:13.193184 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.193667 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.194501 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:13.194707 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:13.194783 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:13.197037 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:13.197040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:13.197041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:13.197061 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:13.198435 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:13.198822 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:13.198825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:13.198826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:13.198895 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:13.199133 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:13.199135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:13.199136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:13.199278 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:13.199372 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:13.199592 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 15:02:13.199682 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 15:02:13.201267 1067 tablet_service.cc:2908] Rejecting scan request for tablet 59cceaaea8eb48669794198ed5f73610: Uninitialized: clean time has not yet been initialized
I20260927 15:02:13.203375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 1024
I20260927 15:02:13.209177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 1158
I20260927 15:02:13.214751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 1292
I20260927 15:02:13.219901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 930
2026-09-27T15:02:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2498 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[1462](info): set up 2 sockets
Sep 27 15:02:13 dist-test-slave-ml7t krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 15:02:14.265534 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:14 dist-test-slave-ml7t krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:14.388307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46823
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:14.503057 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:14.503333 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:14.503376 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:14.507658 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:14.507717 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:02:14.507738 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:14.507759 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:14.507808 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:14.507846 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:14.513175 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46823
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:14.514397 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:14.515497 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:14.524471 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:14.524572 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:14.524572 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:14.524951 1475 server_base.cc:1034] running on GCE node
I20260927 15:02:14.525588 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:14.526780 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:14.527984 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790521334527947 us; error 45 us; skew 500 ppm
Sep 27 15:02:14 dist-test-slave-ml7t krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:14.533931 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:14.534833 1475 webserver.cc:460] Webserver started at http://127.0.98.62:41123/ using document root <none> and password file <none>
I20260927 15:02:14.535339 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:14.535398 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:14.535565 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:14.537540 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c"
format_stamp: "Formatted at 2026-09-27 15:02:14 on dist-test-slave-ml7t"
server_key: "02011feb8a829205f02ef18af8edfa4d"
server_key_iv: "831aa88f70b24c27298101767aa92b46"
server_key_version: "encryptionkey@0"
I20260927 15:02:14.538131 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c"
format_stamp: "Formatted at 2026-09-27 15:02:14 on dist-test-slave-ml7t"
server_key: "02011feb8a829205f02ef18af8edfa4d"
server_key_iv: "831aa88f70b24c27298101767aa92b46"
server_key_version: "encryptionkey@0"
I20260927 15:02:14.541884 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:02:14.544471 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:14.546003 1475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:14.546118 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c"
format_stamp: "Formatted at 2026-09-27 15:02:14 on dist-test-slave-ml7t"
server_key: "02011feb8a829205f02ef18af8edfa4d"
server_key_iv: "831aa88f70b24c27298101767aa92b46"
server_key_version: "encryptionkey@0"
I20260927 15:02:14.546216 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:14.574201 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:14.577502 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:14.577651 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:14.585247 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46823
I20260927 15:02:14.585274 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46823 every 8 connection(s)
I20260927 15:02:14.586265 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:14.589006 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:14.594945 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c: Bootstrap starting.
I20260927 15:02:14.595013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 1475
I20260927 15:02:14.595170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:14.595403 392 external_mini_cluster.cc:1428] Setting key 282b35c1a0a8b82fda04dba0d2c7d067
I20260927 15:02:14.597803 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:14.598713 1553 log.cc:824] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:14.600934 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c: No bootstrap required, opened a new log
I20260927 15:02:14.605201 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46823 } }
Sep 27 15:02:14 dist-test-slave-ml7t krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:14.605506 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:14.605558 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4f8fab89d149ca9bc70ddc7ca89a2c, State: Initialized, Role: FOLLOWER
I20260927 15:02:14.606119 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46823 } }
I20260927 15:02:14.606269 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:14.606355 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:14.606477 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:14.607620 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46823 } }
I20260927 15:02:14.607971 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e4f8fab89d149ca9bc70ddc7ca89a2c; no voters:
I20260927 15:02:14.608274 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:14.608395 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:14.608615 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [term 1 LEADER]: Becoming Leader. State: Replica: 1e4f8fab89d149ca9bc70ddc7ca89a2c, State: Running, Role: LEADER
I20260927 15:02:14.608942 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46823 } }
I20260927 15:02:14.609570 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:14.610610 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46823 } } }
I20260927 15:02:14.610746 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:14.611047 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e4f8fab89d149ca9bc70ddc7ca89a2c. Latest consensus state: current_term: 1 leader_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4f8fab89d149ca9bc70ddc7ca89a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46823 } } }
I20260927 15:02:14.611153 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:14.611403 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:14.612421 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:14.596865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.0.98.62:46823)
0927 15:02:14.597365 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:14.597383 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:14.597425 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:02:14.598356 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:14.598379 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:14.598838 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:14.599219 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:14.600016 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.601444 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.602441 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.603222 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.603396 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:14.606208 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:14.606213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:14.606226 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:14.606291 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:14.608262 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:14.608836 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:14.608842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:14.608847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:14.608985 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:14.609297 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:14.609301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:14.609302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:14.609952 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:14.610139 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:14.610957 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 15:02:14.611301 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":151,"threads_started":1}
I20260927 15:02:14.614586 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:14.620108 1567 catalog_manager.cc:1347] Generated new cluster ID: 1275a7e6b10b45c6a79666f0631467b0
I20260927 15:02:14.620184 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:14.640295 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:14.641292 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:14.658416 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c: Generated new TSK 0
I20260927 15:02:14.659322 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:14.689393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46823
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:14.818040 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:14.818311 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:14.818356 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:14.822402 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:14.822458 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:14.822496 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:14.822582 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:14.827837 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:14.828871 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:14.829917 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:14.839324 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:14.839365 1579 server_base.cc:1034] running on GCE node
W20260927 15:02:14.839366 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:14.839362 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:14.839946 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:14.840569 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:14.841754 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790521334841736 us; error 29 us; skew 500 ppm
Sep 27 15:02:14 dist-test-slave-ml7t krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:14.844704 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:14.845654 1579 webserver.cc:460] Webserver started at http://127.0.98.1:46865/ using document root <none> and password file <none>
I20260927 15:02:14.846268 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:14.846364 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:14.846608 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:14.848649 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "3f95705c47f34a668113e7d248c4405c"
format_stamp: "Formatted at 2026-09-27 15:02:14 on dist-test-slave-ml7t"
server_key: "60b6670a864ec793c7740c3c9aaee1f7"
server_key_iv: "cddfe93f807a3240f24f14bf74e4b5c6"
server_key_version: "encryptionkey@0"
I20260927 15:02:14.849268 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3f95705c47f34a668113e7d248c4405c"
format_stamp: "Formatted at 2026-09-27 15:02:14 on dist-test-slave-ml7t"
server_key: "60b6670a864ec793c7740c3c9aaee1f7"
server_key_iv: "cddfe93f807a3240f24f14bf74e4b5c6"
server_key_version: "encryptionkey@0"
I20260927 15:02:14.853305 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:02:14.855937 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:14.857200 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:14.857348 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "3f95705c47f34a668113e7d248c4405c"
format_stamp: "Formatted at 2026-09-27 15:02:14 on dist-test-slave-ml7t"
server_key: "60b6670a864ec793c7740c3c9aaee1f7"
server_key_iv: "cddfe93f807a3240f24f14bf74e4b5c6"
server_key_version: "encryptionkey@0"
I20260927 15:02:14.857472 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:14.867429 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:14.870308 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:14.870512 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:14.871925 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:14.871997 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:14.872063 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:14.872104 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:02:14 dist-test-slave-ml7t krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:14.883064 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38937
I20260927 15:02:14.883140 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38937 every 8 connection(s)
I20260927 15:02:14.883984 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:14.885963 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:14.872612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39617 (local address 127.0.98.62:46823)
0927 15:02:14.872745 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:14.872749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:14.873764 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 15:02:14.874706 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:14.874709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:14.874764 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:14.874846 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:14.876251 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.876956 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.878354 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.878588 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.878701 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:14.882073 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:14.882077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:14.882079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:14.882101 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:14.883835 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:14.884440 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:14.884446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:14.884449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:14.884539 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:14.884927 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:14.884932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:14.884934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:14.885123 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:14.885207 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:14.885656 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 15:02:14.885816 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:02:14.886322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 1579
I20260927 15:02:14.886438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:14.886689 392 external_mini_cluster.cc:1428] Setting key 4a9c4d20ac64edb9ed5e2616b084cbdd
I20260927 15:02:14.886906 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:14.873155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46823 (local address 127.0.98.1:39617)
0927 15:02:14.873615 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:14.873662 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:14.874501 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:14.875172 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:14.875183 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:14.875592 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:14.876101 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:14.876117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.877084 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.877087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:14.878233 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:14.878242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.878736 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.878740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:14.878930 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:14.879604 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:14.879623 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:14.881861 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:14.883962 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:14.883970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:14.883983 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:14.884306 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:14.884645 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:14.884650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:14.884654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:14.884811 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:14.885219 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:14.885224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:14.885431 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 15:02:14.885583 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 15:02:14.885910 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":140,"threads_started":1}
I20260927 15:02:14.888298 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46823
I20260927 15:02:14.888433 1710 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:14.889005 1710 heartbeater.cc:511] Master 127.0.98.62:46823 requested a full tablet report, sending...
I20260927 15:02:14.890620 1507 ts_manager.cc:194] Registered new tserver with Master: 3f95705c47f34a668113e7d248c4405c (127.0.98.1:38937)
I20260927 15:02:14.891852 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39617
I20260927 15:02:14.897921 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:14.890349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37185 (local address 127.0.98.62:46823)
0927 15:02:14.890469 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:14.890473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:14.890635 (+ 162us) server_negotiation.cc:407] Connection header received
0927 15:02:14.890769 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:14.890772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:14.890819 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:14.890876 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:14.891348 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.891842 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.892983 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.893319 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.893421 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:14.894551 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:14.894556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:14.894558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:14.894584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:14.896228 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:14.896721 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:14.896724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:14.896725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:14.896789 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:14.897097 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:14.897099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:14.897100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:14.897264 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:14.897335 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:14.897645 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:02:14.897726 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":49}
I20260927 15:02:14.898134 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:14.890291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46823 (local address 127.0.98.1:37185)
0927 15:02:14.890549 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:14.890564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:14.890661 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:14.890922 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:14.890925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:14.891129 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:14.891240 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:14.891247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.891972 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.891975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:14.892854 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:14.892868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:14.893444 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:14.893450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:14.893551 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:14.894095 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:14.894109 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:14.894418 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:14.896344 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:14.896351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:14.896355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:14.896625 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:14.896878 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:14.896884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:14.896889 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:14.897004 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:14.897358 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:14.897364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:14.897471 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 15:02:14.897602 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 15:02:14.897742 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:14.917237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46823
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:15.029136 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:15.029386 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:15.029429 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:15.033427 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:15.033486 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:15.033524 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:15.033557 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:15.038744 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:15.039657 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:15.040572 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:15.049716 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:15.049734 1719 server_base.cc:1034] running on GCE node
W20260927 15:02:15.049886 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:15.050084 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:15.050504 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:15.051156 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:15.052349 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790521335052314 us; error 47 us; skew 500 ppm
Sep 27 15:02:15 dist-test-slave-ml7t krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:15.055135 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:15.055889 1719 webserver.cc:460] Webserver started at http://127.0.98.2:35459/ using document root <none> and password file <none>
I20260927 15:02:15.056375 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:15.056428 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:15.056586 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:15.058526 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0f0488caa644335ac54cf4571e46338"
format_stamp: "Formatted at 2026-09-27 15:02:15 on dist-test-slave-ml7t"
server_key: "915b2d28bb51d3bab87990b871419e3a"
server_key_iv: "cd45c15872aa78c4fd4f0d74ad3a7614"
server_key_version: "encryptionkey@0"
I20260927 15:02:15.059019 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0f0488caa644335ac54cf4571e46338"
format_stamp: "Formatted at 2026-09-27 15:02:15 on dist-test-slave-ml7t"
server_key: "915b2d28bb51d3bab87990b871419e3a"
server_key_iv: "cd45c15872aa78c4fd4f0d74ad3a7614"
server_key_version: "encryptionkey@0"
I20260927 15:02:15.062572 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:15.064885 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:15.066254 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:15.066349 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "b0f0488caa644335ac54cf4571e46338"
format_stamp: "Formatted at 2026-09-27 15:02:15 on dist-test-slave-ml7t"
server_key: "915b2d28bb51d3bab87990b871419e3a"
server_key_iv: "cd45c15872aa78c4fd4f0d74ad3a7614"
server_key_version: "encryptionkey@0"
I20260927 15:02:15.066430 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:15.075382 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:15.078542 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:15.078691 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:15.080086 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:15.080137 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:15.080176 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:15.080230 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:02:15 dist-test-slave-ml7t krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:15.091001 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44407
I20260927 15:02:15.091027 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44407 every 8 connection(s)
I20260927 15:02:15.091848 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:15.093490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 1719
I20260927 15:02:15.093604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:15.093979 392 external_mini_cluster.cc:1428] Setting key bb710702917bf9909253ba925b6bb410
I20260927 15:02:15.094816 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.081016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38305 (local address 127.0.98.62:46823)
0927 15:02:15.081174 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.081180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.082012 (+ 832us) server_negotiation.cc:407] Connection header received
0927 15:02:15.083197 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.083204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.083270 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.083363 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:15.085164 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.086102 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.087513 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.087904 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.088019 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.090884 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:15.090888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:15.090890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:15.090917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:15.092591 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.093174 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.093178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.093179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.093266 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.093628 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.093632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.093634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.093898 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:15.094007 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.094559 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.094687 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:02:15.095727 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.081347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46823 (local address 127.0.98.2:38305)
0927 15:02:15.081836 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.081885 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.082935 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.083696 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.083708 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.084211 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:15.084926 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.084952 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.086267 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.086271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.087375 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.087384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.088095 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.088100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.088268 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.088954 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:15.088974 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:15.090659 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:15.092714 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.092721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.092734 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.093044 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.093360 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.093364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.093367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.093503 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.093992 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:15.093997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:15.094262 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.094477 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.094801 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
I20260927 15:02:15.096894 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46823
I20260927 15:02:15.097028 1850 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:15.097589 1850 heartbeater.cc:511] Master 127.0.98.62:46823 requested a full tablet report, sending...
I20260927 15:02:15.098706 1506 ts_manager.cc:194] Registered new tserver with Master: b0f0488caa644335ac54cf4571e46338 (127.0.98.2:44407)
I20260927 15:02:15.099467 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38305
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:15.106796 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.099252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49579 (local address 127.0.98.62:46823)
0927 15:02:15.099343 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.099346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.099358 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:15.099389 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.099392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.099429 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.099515 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:15.100006 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.100844 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.102198 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.102399 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.102483 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.103758 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:15.103761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:15.103763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:15.103783 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:15.105201 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.105647 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.105650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.105651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.105730 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.106058 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.106061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.106062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.106303 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:15.106383 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.106606 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.106690 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:02:15.106799 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.098941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46823 (local address 127.0.98.2:49579)
0927 15:02:15.099183 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.099195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.099260 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.099534 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.099536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.099765 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:15.099886 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.099893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.101004 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.101011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.102066 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.102081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.102518 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.102524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.102661 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.103142 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:15.103158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:15.103616 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:15.105343 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.105349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.105354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.105563 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.105863 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.105865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.105868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.105949 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.106384 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:15.106386 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:15.106456 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.106558 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.106661 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:15.124818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46823
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:15.235880 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:15.236233 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:15.236327 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:15.241547 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:15.241632 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:15.241708 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:15.241797 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:15.247229 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:15.248245 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:15.249310 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:15.259016 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:15.259016 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:15.259148 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:15.259369 1859 server_base.cc:1034] running on GCE node
I20260927 15:02:15.259768 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:15.260424 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:15.261687 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790521335261642 us; error 65 us; skew 500 ppm
Sep 27 15:02:15 dist-test-slave-ml7t krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:15.264571 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:15.265497 1859 webserver.cc:460] Webserver started at http://127.0.98.3:44031/ using document root <none> and password file <none>
I20260927 15:02:15.266160 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:15.266252 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:15.266491 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:15.268465 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "abb3fd20a1d34ec2a598d73df66a3ba3"
format_stamp: "Formatted at 2026-09-27 15:02:15 on dist-test-slave-ml7t"
server_key: "9ee6b399287f49433ae84783cf8220f4"
server_key_iv: "13e7347669e7269f8564b191dc0b09fe"
server_key_version: "encryptionkey@0"
I20260927 15:02:15.269013 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "abb3fd20a1d34ec2a598d73df66a3ba3"
format_stamp: "Formatted at 2026-09-27 15:02:15 on dist-test-slave-ml7t"
server_key: "9ee6b399287f49433ae84783cf8220f4"
server_key_iv: "13e7347669e7269f8564b191dc0b09fe"
server_key_version: "encryptionkey@0"
I20260927 15:02:15.272792 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:02:15.275359 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:15.276604 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 15:02:15.276746 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "abb3fd20a1d34ec2a598d73df66a3ba3"
format_stamp: "Formatted at 2026-09-27 15:02:15 on dist-test-slave-ml7t"
server_key: "9ee6b399287f49433ae84783cf8220f4"
server_key_iv: "13e7347669e7269f8564b191dc0b09fe"
server_key_version: "encryptionkey@0"
I20260927 15:02:15.276853 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:15.299438 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:15.302809 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:15.303092 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:15.304507 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:15.304586 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:15.304673 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:15.304723 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:02:15 dist-test-slave-ml7t krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:15.315804 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36889
I20260927 15:02:15.315898 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36889 every 8 connection(s)
I20260927 15:02:15.316843 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:15.318889 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.305516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42285 (local address 127.0.98.62:46823)
0927 15:02:15.305731 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.305737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.306770 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 15:02:15.307983 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.307989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.308052 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.308156 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:15.309721 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.310660 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.312072 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.312471 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.312601 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.315228 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:15.315231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:15.315234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:15.315258 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:15.316826 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.317423 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.317426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.317428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.317538 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.317915 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.317917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.317918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.318136 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:15.318219 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.318672 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.318770 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 15:02:15.319682 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.306123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46823 (local address 127.0.98.3:42285)
0927 15:02:15.306621 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.306663 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.307704 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.308367 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.308379 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.308777 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:15.309554 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.309575 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.310783 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.310786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.311928 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.311943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.312631 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.312637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.312824 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.313402 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:15.313416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:15.315046 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:15.316943 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.316948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.316962 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.317303 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.317646 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.317649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.317651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.317832 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.318227 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:15.318232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:15.318441 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.318609 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.318847 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":134,"threads_started":1}
I20260927 15:02:15.321105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 1859
I20260927 15:02:15.321251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:15.321523 392 external_mini_cluster.cc:1428] Setting key b4cc99b30255636910c26da9e5a80ade
I20260927 15:02:15.322937 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46823
I20260927 15:02:15.323066 1990 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:15.323554 1990 heartbeater.cc:511] Master 127.0.98.62:46823 requested a full tablet report, sending...
I20260927 15:02:15.324498 1507 ts_manager.cc:194] Registered new tserver with Master: abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889)
I20260927 15:02:15.325048 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42285
I20260927 15:02:15.330384 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.323568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50909 (local address 127.0.98.62:46823)
0927 15:02:15.323686 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.323691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.323703 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:15.323738 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.323741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.323779 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.323864 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:15.324276 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.324857 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.325733 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.326018 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.326136 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.327307 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:15.327311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:15.327313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:15.327336 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:15.328770 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.329192 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.329197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.329199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.329275 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.329554 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.329558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.329560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.329725 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:15.329875 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.330127 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.330229 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:02:15.330394 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.323158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46823 (local address 127.0.98.3:50909)
0927 15:02:15.323421 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.323438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.323537 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.323902 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.323904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.324070 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:15.324159 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.324165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.324975 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.324980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.325613 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.325621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.326147 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.326151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.326240 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.326756 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:15.326770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:15.327179 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:15.328897 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.328900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.328902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.329093 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.329372 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.329374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.329376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.329466 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.329862 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:15.329865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:15.329956 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.330067 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.330202 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
I20260927 15:02:15.335359 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:02:15 dist-test-slave-ml7t krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:02:15 dist-test-slave-ml7t krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:15.360585 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.352240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.0.98.62:46823)
0927 15:02:15.352369 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.352372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.352386 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:15.352426 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.352428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.352471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.352550 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:15.353308 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.353897 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.354755 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.355000 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.355084 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.357519 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:15.357522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:15.357524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:15.357549 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:15.358963 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.359369 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.359372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.359373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.359447 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.359756 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.359758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.359759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.359923 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:15.360022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.360263 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.360355 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:02:15.363842 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:15.367027 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:15.381194 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.376258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.98.1:38937)
0927 15:02:15.376814 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.376829 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.376854 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:02:15.376946 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.376954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.377156 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.377314 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:15.377602 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.378438 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.379837 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.380531 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.380645 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.380800 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.380956 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.381049 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":96,"threads_started":1}
I20260927 15:02:15.381215 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.375532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38937 (local address 127.0.0.1:45748)
0927 15:02:15.376319 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.376374 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.376504 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.377315 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.377321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.377341 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:15.377478 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.377485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.378562 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.378569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.379717 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.379725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.380653 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.380656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.380761 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.380803 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.380916 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.381032 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"spinlock_wait_cycles":5248,"thread_start_us":89,"threads_started":1}
I20260927 15:02:15.381657 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.375532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44407 (local address 127.0.0.1:41320)
0927 15:02:15.376318 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.376371 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.376503 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.377545 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.377548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.377563 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:15.377656 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.377664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.379133 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.379137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.380125 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.380134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.381276 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.381281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.381448 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.381466 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.381509 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.381558 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":200,"spinlock_wait_cycles":5248,"thread_start_us":89,"threads_started":1}
I20260927 15:02:15.382411 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.375683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.98.2:44407)
0927 15:02:15.377163 (+ 1480us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.377173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.377192 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:02:15.377252 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.377255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.377406 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.377821 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:15.377942 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.378994 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.380278 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.381069 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.381949 (+ 880us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.382123 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.382174 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.382269 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":160,"threads_started":1}
I20260927 15:02:15.383394 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.375786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36889 (local address 127.0.0.1:52894)
0927 15:02:15.376839 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.376852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.376928 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.377484 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.377486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.377500 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:15.377592 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.377597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.379133 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.379137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.382002 (+ 2865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.382011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.382985 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.382989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.383086 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.383114 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.383205 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.383267 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260927 15:02:15.383777 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.376702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.98.3:36889)
0927 15:02:15.377008 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.377020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.377044 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:02:15.377116 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.377121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.377305 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.377452 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:15.377691 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.378884 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.382137 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.382858 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.382984 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.383127 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.383548 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.383644 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":134,"threads_started":1}
I20260927 15:02:15.384534 1644 tablet_service.cc:1467] Processing CreateTablet for tablet d4d77183b9ca4d6f8ba870ca38d011eb (DEFAULT_TABLE table=test-table [id=4c34ae19611140f0bc1ee8e8d5cea348]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:15.384953 1784 tablet_service.cc:1467] Processing CreateTablet for tablet d4d77183b9ca4d6f8ba870ca38d011eb (DEFAULT_TABLE table=test-table [id=4c34ae19611140f0bc1ee8e8d5cea348]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:15.385699 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d77183b9ca4d6f8ba870ca38d011eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.386003 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d77183b9ca4d6f8ba870ca38d011eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.386310 1924 tablet_service.cc:1467] Processing CreateTablet for tablet d4d77183b9ca4d6f8ba870ca38d011eb (DEFAULT_TABLE table=test-table [id=4c34ae19611140f0bc1ee8e8d5cea348]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:15.387286 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d77183b9ca4d6f8ba870ca38d011eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.392709 2013 tablet_bootstrap.cc:492] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Bootstrap starting.
I20260927 15:02:15.394088 2014 tablet_bootstrap.cc:492] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Bootstrap starting.
I20260927 15:02:15.395051 2013 tablet_bootstrap.cc:654] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.395722 2015 tablet_bootstrap.cc:492] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Bootstrap starting.
I20260927 15:02:15.395955 2013 log.cc:824] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:15.396832 2014 tablet_bootstrap.cc:654] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.397647 2014 log.cc:824] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:15.397755 2015 tablet_bootstrap.cc:654] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.398260 2013 tablet_bootstrap.cc:492] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: No bootstrap required, opened a new log
I20260927 15:02:15.398488 2013 ts_tablet_manager.cc:1397] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:02:15.398582 2015 log.cc:824] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:15.399878 2014 tablet_bootstrap.cc:492] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: No bootstrap required, opened a new log
I20260927 15:02:15.400058 2014 ts_tablet_manager.cc:1397] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:02:15.400170 2015 tablet_bootstrap.cc:492] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: No bootstrap required, opened a new log
I20260927 15:02:15.400339 2015 ts_tablet_manager.cc:1397] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:02:15.402462 2013 raft_consensus.cc:348] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.402725 2013 raft_consensus.cc:374] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.402783 2013 raft_consensus.cc:729] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f0488caa644335ac54cf4571e46338, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.403229 2013 consensus_queue.cc:260] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.403905 1850 heartbeater.cc:503] Master 127.0.98.62:46823 was elected leader, sending a full tablet report...
I20260927 15:02:15.404031 2014 raft_consensus.cc:348] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.404263 2014 raft_consensus.cc:374] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.404320 2013 ts_tablet_manager.cc:1428] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:02:15.404318 2014 raft_consensus.cc:729] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f95705c47f34a668113e7d248c4405c, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.404706 2014 consensus_queue.cc:260] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.403584 2015 raft_consensus.cc:348] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.404899 2015 raft_consensus.cc:374] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.404969 2015 raft_consensus.cc:729] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3fd20a1d34ec2a598d73df66a3ba3, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.405232 1710 heartbeater.cc:503] Master 127.0.98.62:46823 was elected leader, sending a full tablet report...
I20260927 15:02:15.405352 2015 consensus_queue.cc:260] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.405614 2014 ts_tablet_manager.cc:1428] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:02:15.406306 2015 ts_tablet_manager.cc:1428] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:02:15.406629 1990 heartbeater.cc:503] Master 127.0.98.62:46823 was elected leader, sending a full tablet report...
I20260927 15:02:15.552867 2019 raft_consensus.cc:482] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:15.553069 2019 raft_consensus.cc:504] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.554452 2019 leader_election.cc:288] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f95705c47f34a668113e7d248c4405c (127.0.98.1:38937), abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889)
I20260927 15:02:15.558496 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.554758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36889 (local address 127.0.98.2:33235)
0927 15:02:15.554875 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.554892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.554994 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.555431 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.555434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.555460 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:15.555601 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.555611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.556419 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.556422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.557376 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.557388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.558034 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.558038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.558206 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.558241 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.558324 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.558385 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":52}
I20260927 15:02:15.558531 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.554981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33235 (local address 127.0.98.3:36889)
0927 15:02:15.555172 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.555176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.555193 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:02:15.555252 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.555258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.555310 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.555399 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:15.555723 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.556304 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.557499 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.557916 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.557995 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.558055 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.558340 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.558410 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 15:02:15.559257 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d77183b9ca4d6f8ba870ca38d011eb" candidate_uuid: "b0f0488caa644335ac54cf4571e46338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" is_pre_election: true
I20260927 15:02:15.559461 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.554821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60775 (local address 127.0.98.1:38937)
0927 15:02:15.554954 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.554959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.555573 (+ 614us) server_negotiation.cc:407] Connection header received
0927 15:02:15.555651 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.555653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.555731 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.555812 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:15.556479 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.556974 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.558335 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.558922 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.559026 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.559085 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.559285 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.559337 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:02:15.559476 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.554758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38937 (local address 127.0.98.2:60775)
0927 15:02:15.555434 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.555447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.555569 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.555828 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.555830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.555844 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:15.556316 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.556322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.557088 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.557091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.558191 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.558200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.559068 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.559072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.559163 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.559185 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.559268 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.559337 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":381,"thread_start_us":145,"threads_started":1}
I20260927 15:02:15.559633 1944 raft_consensus.cc:2454] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f0488caa644335ac54cf4571e46338 in term 0.
I20260927 15:02:15.560176 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d77183b9ca4d6f8ba870ca38d011eb" candidate_uuid: "b0f0488caa644335ac54cf4571e46338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f95705c47f34a668113e7d248c4405c" is_pre_election: true
I20260927 15:02:15.560232 1737 leader_election.cc:302] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abb3fd20a1d34ec2a598d73df66a3ba3, b0f0488caa644335ac54cf4571e46338; no voters:
I20260927 15:02:15.560456 1664 raft_consensus.cc:2454] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f0488caa644335ac54cf4571e46338 in term 0.
I20260927 15:02:15.560528 2019 raft_consensus.cc:2781] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:15.560608 2019 raft_consensus.cc:482] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:15.560662 2019 raft_consensus.cc:3037] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.561806 2019 raft_consensus.cc:504] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.562191 2019 leader_election.cc:288] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [CANDIDATE]: Term 1 election: Requested vote from peers 3f95705c47f34a668113e7d248c4405c (127.0.98.1:38937), abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889)
I20260927 15:02:15.562482 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d77183b9ca4d6f8ba870ca38d011eb" candidate_uuid: "b0f0488caa644335ac54cf4571e46338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f95705c47f34a668113e7d248c4405c"
I20260927 15:02:15.562592 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d77183b9ca4d6f8ba870ca38d011eb" candidate_uuid: "b0f0488caa644335ac54cf4571e46338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3"
I20260927 15:02:15.562608 1664 raft_consensus.cc:3037] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.562721 1944 raft_consensus.cc:3037] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.563630 1664 raft_consensus.cc:2454] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f0488caa644335ac54cf4571e46338 in term 1.
I20260927 15:02:15.563911 1738 leader_election.cc:302] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f95705c47f34a668113e7d248c4405c, b0f0488caa644335ac54cf4571e46338; no voters:
I20260927 15:02:15.564064 2019 raft_consensus.cc:2781] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:15.564225 1944 raft_consensus.cc:2454] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f0488caa644335ac54cf4571e46338 in term 1.
I20260927 15:02:15.564289 2019 raft_consensus.cc:686] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 1 LEADER]: Becoming Leader. State: Replica: b0f0488caa644335ac54cf4571e46338, State: Running, Role: LEADER
I20260927 15:02:15.564637 2019 consensus_queue.cc:237] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
W20260927 15:02:15.567759 1991 tablet.cc:2369] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:15.567896 1505 catalog_manager.cc:5495] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0f0488caa644335ac54cf4571e46338 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b0f0488caa644335ac54cf4571e46338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } health_report { overall_health: UNKNOWN } } }
W20260927 15:02:15.592999 1851 tablet.cc:2369] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:15.624212 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.620446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.0.98.2:44407)
0927 15:02:15.620603 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.620607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.620621 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:15.620781 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.620783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.620838 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.620925 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:15.621266 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.621832 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.622719 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.622978 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.623056 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.623389 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:15.623779 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:15.623862 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.623993 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.624039 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:02:15.629863 1944 raft_consensus.cc:1261] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer b0f0488caa644335ac54cf4571e46338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:15.629882 1664 raft_consensus.cc:1261] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Refusing update from remote peer b0f0488caa644335ac54cf4571e46338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:15.630633 2019 consensus_queue.cc:1035] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:15.630857 2023 consensus_queue.cc:1035] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:02:15.636165 1711 tablet.cc:2369] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:15.637639 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333975390732505088 to 7333975390475743232. Current Snapshot: MvccSnapshot[applied={T|T < 7333975390732505088}]
I20260927 15:02:15.640121 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333975390732505088 to 7333975390475743232. Current Snapshot: MvccSnapshot[applied={T|T < 7333975390732505088}]
I20260927 15:02:15.651350 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:15.651546 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:15.654119 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c: Sending DeleteTablet for 3 replicas of tablet d4d77183b9ca4d6f8ba870ca38d011eb
I20260927 15:02:15.654860 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d77183b9ca4d6f8ba870ca38d011eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41320
I20260927 15:02:15.654879 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d77183b9ca4d6f8ba870ca38d011eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45748
I20260927 15:02:15.654879 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d77183b9ca4d6f8ba870ca38d011eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52894
I20260927 15:02:15.655601 2038 tablet_replica.cc:330] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: stopping tablet replica
I20260927 15:02:15.655622 2036 tablet_replica.cc:330] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: stopping tablet replica
I20260927 15:02:15.655965 2038 raft_consensus.cc:2229] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:15.655964 2036 raft_consensus.cc:2229] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:15.656286 2038 raft_consensus.cc:2258] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:15.656330 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:02:15.656663 2037 tablet_replica.cc:330] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: stopping tablet replica
W20260927 15:02:15.656751 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:15.656904 2036 raft_consensus.cc:2258] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:15.656970 2037 raft_consensus.cc:2229] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:15.657269 2037 raft_consensus.cc:2258] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:15.658645 2036 ts_tablet_manager.cc:1905] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:15.659750 2038 ts_tablet_manager.cc:1905] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:15.661547 2036 ts_tablet_manager.cc:1918] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:15.661662 2036 log.cc:1196] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/d4d77183b9ca4d6f8ba870ca38d011eb
I20260927 15:02:15.662082 2036 ts_tablet_manager.cc:1939] T d4d77183b9ca4d6f8ba870ca38d011eb P b0f0488caa644335ac54cf4571e46338: Deleting consensus metadata
I20260927 15:02:15.662056 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9d624a6caa4798a66d8583aae59580 (DEFAULT_TABLE table=test-table [id=e701210e8b8b49298202dc6ae2c1b8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:15.662323 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9d624a6caa4798a66d8583aae59580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.662557 2038 ts_tablet_manager.cc:1918] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:15.662655 2038 log.cc:1196] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/d4d77183b9ca4d6f8ba870ca38d011eb
I20260927 15:02:15.662972 2038 ts_tablet_manager.cc:1939] T d4d77183b9ca4d6f8ba870ca38d011eb P 3f95705c47f34a668113e7d248c4405c: Deleting consensus metadata
I20260927 15:02:15.664100 1494 catalog_manager.cc:4841] TS 3f95705c47f34a668113e7d248c4405c (127.0.98.1:38937): tablet d4d77183b9ca4d6f8ba870ca38d011eb (table test-table [id=4c34ae19611140f0bc1ee8e8d5cea348]) successfully deleted
I20260927 15:02:15.664386 2014 tablet_bootstrap.cc:492] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: Bootstrap starting.
I20260927 15:02:15.664386 2037 ts_tablet_manager.cc:1905] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:15.665313 2014 tablet_bootstrap.cc:654] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.666384 1491 catalog_manager.cc:4841] TS b0f0488caa644335ac54cf4571e46338 (127.0.98.2:44407): tablet d4d77183b9ca4d6f8ba870ca38d011eb (table test-table [id=4c34ae19611140f0bc1ee8e8d5cea348]) successfully deleted
I20260927 15:02:15.666792 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9d624a6caa4798a66d8583aae59580 (DEFAULT_TABLE table=test-table [id=e701210e8b8b49298202dc6ae2c1b8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:15.666949 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9d624a6caa4798a66d8583aae59580 (DEFAULT_TABLE table=test-table [id=e701210e8b8b49298202dc6ae2c1b8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:15.667064 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9d624a6caa4798a66d8583aae59580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.667227 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9d624a6caa4798a66d8583aae59580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.667243 2037 ts_tablet_manager.cc:1918] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:15.667331 2037 log.cc:1196] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/d4d77183b9ca4d6f8ba870ca38d011eb
I20260927 15:02:15.667663 2037 ts_tablet_manager.cc:1939] T d4d77183b9ca4d6f8ba870ca38d011eb P abb3fd20a1d34ec2a598d73df66a3ba3: Deleting consensus metadata
I20260927 15:02:15.669253 2015 tablet_bootstrap.cc:492] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: Bootstrap starting.
I20260927 15:02:15.669351 2013 tablet_bootstrap.cc:492] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: Bootstrap starting.
I20260927 15:02:15.669955 2014 tablet_bootstrap.cc:492] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: No bootstrap required, opened a new log
I20260927 15:02:15.670034 2014 ts_tablet_manager.cc:1397] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:02:15.670269 2015 tablet_bootstrap.cc:654] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.670269 2013 tablet_bootstrap.cc:654] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.670463 2014 raft_consensus.cc:348] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.670571 2014 raft_consensus.cc:374] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.670604 2014 raft_consensus.cc:729] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f95705c47f34a668113e7d248c4405c, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.670720 2014 consensus_queue.cc:260] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.670950 2014 ts_tablet_manager.cc:1428] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:15.671058 1493 catalog_manager.cc:4841] TS abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889): tablet d4d77183b9ca4d6f8ba870ca38d011eb (table test-table [id=4c34ae19611140f0bc1ee8e8d5cea348]) successfully deleted
I20260927 15:02:15.671851 2013 tablet_bootstrap.cc:492] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: No bootstrap required, opened a new log
I20260927 15:02:15.671937 2013 ts_tablet_manager.cc:1397] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:02:15.672111 2015 tablet_bootstrap.cc:492] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: No bootstrap required, opened a new log
I20260927 15:02:15.672176 2015 ts_tablet_manager.cc:1397] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:15.672405 2013 raft_consensus.cc:348] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.672498 2013 raft_consensus.cc:374] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.672531 2013 raft_consensus.cc:729] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f0488caa644335ac54cf4571e46338, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.672598 2015 raft_consensus.cc:348] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.672689 2015 raft_consensus.cc:374] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.672650 2013 consensus_queue.cc:260] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.672727 2015 raft_consensus.cc:729] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3fd20a1d34ec2a598d73df66a3ba3, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.672911 2013 ts_tablet_manager.cc:1428] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:15.672853 2015 consensus_queue.cc:260] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.673136 2015 ts_tablet_manager.cc:1428] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:15.842849 2020 raft_consensus.cc:482] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:15.842993 2020 raft_consensus.cc:504] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.844272 2020 leader_election.cc:288] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f0488caa644335ac54cf4571e46338 (127.0.98.2:44407), abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889)
I20260927 15:02:15.848519 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.844692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49183 (local address 127.0.98.2:44407)
0927 15:02:15.844841 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.844846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.845167 (+ 321us) server_negotiation.cc:407] Connection header received
0927 15:02:15.845317 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.845320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.845368 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.845448 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:15.845720 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.846264 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.847504 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.847883 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.848034 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.848100 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.848331 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.848396 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:15.848529 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.844730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58063 (local address 127.0.98.3:36889)
0927 15:02:15.844866 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.844871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.845080 (+ 209us) server_negotiation.cc:407] Connection header received
0927 15:02:15.845244 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.845252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.845291 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.845376 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:15.845638 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.846228 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.847497 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.847886 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.848007 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.848073 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.848321 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.848389 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:15.848567 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.844574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36889 (local address 127.0.98.1:58063)
0927 15:02:15.844969 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.844981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.845138 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.845387 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.845390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.845406 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:15.845532 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.845537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.846334 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.846341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.847376 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.847385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.848032 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.848036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.848188 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.848215 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.848306 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.848377 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":104,"thread_start_us":100,"threads_started":1}
I20260927 15:02:15.848567 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.844574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44407 (local address 127.0.98.1:49183)
0927 15:02:15.845063 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.845075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.845204 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.845460 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.845463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.845477 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:15.845609 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.845614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.846372 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.846379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.847376 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.847384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.848038 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.848041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.848214 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.848230 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.848316 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.848384 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":152,"thread_start_us":120,"threads_started":1}
I20260927 15:02:15.849052 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9d624a6caa4798a66d8583aae59580" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" is_pre_election: true
I20260927 15:02:15.849231 1944 raft_consensus.cc:2454] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 0.
I20260927 15:02:15.849171 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9d624a6caa4798a66d8583aae59580" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f0488caa644335ac54cf4571e46338" is_pre_election: true
I20260927 15:02:15.849414 1804 raft_consensus.cc:2454] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 0.
I20260927 15:02:15.849606 1597 leader_election.cc:302] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f95705c47f34a668113e7d248c4405c, abb3fd20a1d34ec2a598d73df66a3ba3; no voters:
I20260927 15:02:15.849880 2020 raft_consensus.cc:2781] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:15.849934 2020 raft_consensus.cc:482] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:15.849964 2020 raft_consensus.cc:3037] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.851066 2020 raft_consensus.cc:504] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.851409 2020 leader_election.cc:288] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 election: Requested vote from peers b0f0488caa644335ac54cf4571e46338 (127.0.98.2:44407), abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889)
I20260927 15:02:15.851738 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9d624a6caa4798a66d8583aae59580" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3"
I20260927 15:02:15.851862 1944 raft_consensus.cc:3037] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.852733 1944 raft_consensus.cc:2454] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 1.
I20260927 15:02:15.852975 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9d624a6caa4798a66d8583aae59580" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f0488caa644335ac54cf4571e46338"
I20260927 15:02:15.853034 1597 leader_election.cc:302] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f95705c47f34a668113e7d248c4405c, abb3fd20a1d34ec2a598d73df66a3ba3; no voters:
I20260927 15:02:15.853111 1804 raft_consensus.cc:3037] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.853202 2020 raft_consensus.cc:2781] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:15.853420 2020 raft_consensus.cc:686] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 1 LEADER]: Becoming Leader. State: Replica: 3f95705c47f34a668113e7d248c4405c, State: Running, Role: LEADER
I20260927 15:02:15.853669 2020 consensus_queue.cc:237] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.854310 1804 raft_consensus.cc:2454] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 1.
I20260927 15:02:15.856226 1506 catalog_manager.cc:5495] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f95705c47f34a668113e7d248c4405c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:02:15 dist-test-slave-ml7t krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:15.896219 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.887588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.0.98.62:46823)
0927 15:02:15.887803 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.887807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.887919 (+ 112us) server_negotiation.cc:407] Connection header received
0927 15:02:15.888133 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.888136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.888178 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.888273 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:15.888963 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.889524 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.890334 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.890554 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.890662 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.893002 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:15.893005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:15.893010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:15.893034 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:15.894535 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.895004 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.895007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.895009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.895083 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:15.895355 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:15.895358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:15.895360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:15.895528 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:15.895628 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.895913 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.896080 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260927 15:02:15.896219 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.887510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46823 (local address 127.0.0.1:38916)
0927 15:02:15.887811 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.887827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.888024 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.888297 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.888300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.888737 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:15.888828 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.888834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.889629 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.889636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.890221 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.890228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.890661 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.890664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.890737 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.891339 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:15.891351 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:15.892862 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:15.894664 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.894672 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.894678 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.894892 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.895172 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:15.895176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:15.895178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:15.895260 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:15.895635 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:15.895644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:15.895752 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.895862 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.896022 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":69,"threads_started":1}
I20260927 15:02:15.900570 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38916:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:02:15.901381 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:15.906562 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 15d85ccd671f46b683ff90feb8cbf0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b12202a24ea94cf2b6f37fb73faab734]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:15.906764 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 15d85ccd671f46b683ff90feb8cbf0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b12202a24ea94cf2b6f37fb73faab734]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:15.906967 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d85ccd671f46b683ff90feb8cbf0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.906834 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 15d85ccd671f46b683ff90feb8cbf0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b12202a24ea94cf2b6f37fb73faab734]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:15.907085 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d85ccd671f46b683ff90feb8cbf0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.907164 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d85ccd671f46b683ff90feb8cbf0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:15.909489 2013 tablet_bootstrap.cc:492] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: Bootstrap starting.
I20260927 15:02:15.909813 2015 tablet_bootstrap.cc:492] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: Bootstrap starting.
I20260927 15:02:15.910483 2013 tablet_bootstrap.cc:654] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.910615 2015 tablet_bootstrap.cc:654] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.911482 2014 tablet_bootstrap.cc:492] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: Bootstrap starting.
I20260927 15:02:15.911628 2013 tablet_bootstrap.cc:492] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: No bootstrap required, opened a new log
I20260927 15:02:15.911695 2013 ts_tablet_manager.cc:1397] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:15.912124 2013 raft_consensus.cc:348] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.912250 2013 raft_consensus.cc:374] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.912294 2014 tablet_bootstrap.cc:654] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:15.912290 2013 raft_consensus.cc:729] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f0488caa644335ac54cf4571e46338, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.912429 2013 consensus_queue.cc:260] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.912737 2013 ts_tablet_manager.cc:1428] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:02:15.912699 2023 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.912824 2023 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:15.913265 2015 tablet_bootstrap.cc:492] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: No bootstrap required, opened a new log
I20260927 15:02:15.913372 2015 ts_tablet_manager.cc:1397] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:02:15.913703 2014 tablet_bootstrap.cc:492] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: No bootstrap required, opened a new log
I20260927 15:02:15.913810 2014 ts_tablet_manager.cc:1397] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:15.914129 2015 raft_consensus.cc:348] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.914250 2015 raft_consensus.cc:374] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.914306 2015 raft_consensus.cc:729] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3fd20a1d34ec2a598d73df66a3ba3, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.914273 2014 raft_consensus.cc:348] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.914400 2014 raft_consensus.cc:374] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:15.914438 2014 raft_consensus.cc:729] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f95705c47f34a668113e7d248c4405c, State: Initialized, Role: FOLLOWER
I20260927 15:02:15.914430 2015 consensus_queue.cc:260] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.914572 2014 consensus_queue.cc:260] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.914685 2015 ts_tablet_manager.cc:1428] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:15.914830 2014 ts_tablet_manager.cc:1428] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:02:15.914775 2020 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.914909 2020 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:15.914858 2021 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.915009 2021 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:15.953809 2020 raft_consensus.cc:482] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:15.953938 2020 raft_consensus.cc:504] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.954330 2020 leader_election.cc:288] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f0488caa644335ac54cf4571e46338 (127.0.98.2:44407), abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889)
I20260927 15:02:15.954761 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d85ccd671f46b683ff90feb8cbf0b9" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" is_pre_election: true
I20260927 15:02:15.954761 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d85ccd671f46b683ff90feb8cbf0b9" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f0488caa644335ac54cf4571e46338" is_pre_election: true
I20260927 15:02:15.954913 1944 raft_consensus.cc:2454] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 0.
I20260927 15:02:15.954913 1804 raft_consensus.cc:2454] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 0.
I20260927 15:02:15.955204 1597 leader_election.cc:302] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f95705c47f34a668113e7d248c4405c, abb3fd20a1d34ec2a598d73df66a3ba3; no voters:
I20260927 15:02:15.955389 2020 raft_consensus.cc:2781] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:15.955462 2020 raft_consensus.cc:482] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:15.955490 2020 raft_consensus.cc:3037] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.956485 2020 raft_consensus.cc:504] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.956816 2020 leader_election.cc:288] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 election: Requested vote from peers b0f0488caa644335ac54cf4571e46338 (127.0.98.2:44407), abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889)
I20260927 15:02:15.957149 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d85ccd671f46b683ff90feb8cbf0b9" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3"
I20260927 15:02:15.957176 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d85ccd671f46b683ff90feb8cbf0b9" candidate_uuid: "3f95705c47f34a668113e7d248c4405c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f0488caa644335ac54cf4571e46338"
I20260927 15:02:15.957280 1944 raft_consensus.cc:3037] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.957301 1804 raft_consensus.cc:3037] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:15.958251 1804 raft_consensus.cc:2454] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 1.
I20260927 15:02:15.958283 1944 raft_consensus.cc:2454] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f95705c47f34a668113e7d248c4405c in term 1.
I20260927 15:02:15.958556 1595 leader_election.cc:302] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f95705c47f34a668113e7d248c4405c, b0f0488caa644335ac54cf4571e46338; no voters:
I20260927 15:02:15.958765 2020 raft_consensus.cc:2781] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:15.958894 2020 raft_consensus.cc:686] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [term 1 LEADER]: Becoming Leader. State: Replica: 3f95705c47f34a668113e7d248c4405c, State: Running, Role: LEADER
I20260927 15:02:15.959060 2020 consensus_queue.cc:237] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } }
I20260927 15:02:15.959748 2044 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: TxnStatusTablet state changed. Reason: New leader 3f95705c47f34a668113e7d248c4405c. Latest consensus state: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.959842 2044 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:15.960412 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:02:15.960587 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:02:15.960839 1506 catalog_manager.cc:5495] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f95705c47f34a668113e7d248c4405c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } health_report { overall_health: UNKNOWN } } }
I20260927 15:02:15.984447 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.980116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38937 (local address 127.0.0.1:45750)
0927 15:02:15.980378 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:15.980392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:15.980502 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:15.980877 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:15.980880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:15.980889 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:15.981059 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.981065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.982126 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.982129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.982727 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:15.982735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.983321 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.983323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:15.983384 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.983426 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:15.984207 (+ 781us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:15.984217 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 15:02:15.984265 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:02:15.984330 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 15:02:15.984510 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.980231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.0.98.1:38937)
0927 15:02:15.980387 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.980392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.980446 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:02:15.980647 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.980650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.980739 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.980856 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:15.981199 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.982020 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.982848 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.983206 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:15.983321 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:15.983568 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:15.984099 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:15.984218 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:15.984305 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 15:02:15.984364 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:02:15.987335 1944 raft_consensus.cc:1261] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 3f95705c47f34a668113e7d248c4405c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:15.987471 1804 raft_consensus.cc:1261] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Refusing update from remote peer 3f95705c47f34a668113e7d248c4405c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:15.987875 2044 consensus_queue.cc:1035] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:15.988099 2020 consensus_queue.cc:1035] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:15.989447 2021 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: TxnStatusTablet state changed. Reason: New leader 3f95705c47f34a668113e7d248c4405c. Latest consensus state: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.989565 2021 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:15.989657 2023 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: TxnStatusTablet state changed. Reason: New leader 3f95705c47f34a668113e7d248c4405c. Latest consensus state: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.989753 2023 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:15.991133 2020 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.991243 2020 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:15.991825 2020 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.991940 2020 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P 3f95705c47f34a668113e7d248c4405c: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:15.992468 2023 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.992592 2023 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P b0f0488caa644335ac54cf4571e46338: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:15.993650 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333975392198828032 to 7333975392089628672. Current Snapshot: MvccSnapshot[applied={T|T < 7333975392198828032 or (T in {7333975392198828032})}]
I20260927 15:02:15.995198 2021 tablet_replica.cc:439] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f95705c47f34a668113e7d248c4405c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f95705c47f34a668113e7d248c4405c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 } } peers { permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 } } }
I20260927 15:02:15.995303 2021 tablet_replica.cc:442] T 15d85ccd671f46b683ff90feb8cbf0b9 P abb3fd20a1d34ec2a598d73df66a3ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:16.001362 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:15.997733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.0.98.1:38937)
0927 15:02:15.997926 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:15.997932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:15.998132 (+ 200us) server_negotiation.cc:407] Connection header received
0927 15:02:15.998281 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:15.998285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:15.998339 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:15.998422 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:15.998678 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:15.999445 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.000260 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.000600 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.000715 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:16.000917 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:16.000999 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:16.001079 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:16.001164 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 15:02:16.001215 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 15:02:16.013022 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:16.008356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35497 (local address 127.0.98.1:38937)
0927 15:02:16.008484 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:16.008489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:16.009171 (+ 682us) server_negotiation.cc:407] Connection header received
0927 15:02:16.009351 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:16.009354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:16.009400 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:16.009469 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:16.009818 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.010605 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.011950 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.012151 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.012229 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:16.012555 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:16.012613 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:16.012665 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:16.012801 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:02:16.012858 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":50}
I20260927 15:02:16.013101 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:16.008256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38937 (local address 127.0.98.1:35497)
0927 15:02:16.009062 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:16.009079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:16.009224 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:16.009508 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:16.009512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:16.009520 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:16.009662 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:16.009669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.010710 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.010714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:16.011825 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:16.011837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.012261 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.012265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:16.012366 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:16.012422 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:16.012695 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:16.012702 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:02:16.012778 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:02:16.012892 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":22,"thread_start_us":590,"threads_started":1}
I20260927 15:02:16.020218 1944 raft_consensus.cc:1261] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 3f95705c47f34a668113e7d248c4405c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:16.020601 1804 raft_consensus.cc:1261] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Refusing update from remote peer 3f95705c47f34a668113e7d248c4405c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:16.020655 2044 consensus_queue.cc:1035] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3fd20a1d34ec2a598d73df66a3ba3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:16.021041 2044 consensus_queue.cc:1035] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f0488caa644335ac54cf4571e46338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:16.056162 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:16.056308 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:16.058252 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e4f8fab89d149ca9bc70ddc7ca89a2c: Sending DeleteTablet for 3 replicas of tablet 8a9d624a6caa4798a66d8583aae59580
I20260927 15:02:16.058734 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9d624a6caa4798a66d8583aae59580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45748
I20260927 15:02:16.058776 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9d624a6caa4798a66d8583aae59580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41320
I20260927 15:02:16.058867 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9d624a6caa4798a66d8583aae59580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52894
I20260927 15:02:16.058936 2038 tablet_replica.cc:330] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: stopping tablet replica
I20260927 15:02:16.059036 2038 raft_consensus.cc:2229] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:16.059053 2037 tablet_replica.cc:330] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: stopping tablet replica
I20260927 15:02:16.059150 2037 raft_consensus.cc:2229] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:16.059247 2037 raft_consensus.cc:2258] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:16.059310 2038 raft_consensus.cc:2258] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:16.059476 2036 tablet_replica.cc:330] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: stopping tablet replica
I20260927 15:02:16.059636 2036 raft_consensus.cc:2229] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:16.059751 2036 raft_consensus.cc:2258] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:16.060060 2037 ts_tablet_manager.cc:1905] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:16.060518 2036 ts_tablet_manager.cc:1905] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:16.063117 2037 ts_tablet_manager.cc:1918] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:16.063200 2037 log.cc:1196] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/8a9d624a6caa4798a66d8583aae59580
I20260927 15:02:16.063316 2036 ts_tablet_manager.cc:1918] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:16.063382 2036 log.cc:1196] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/8a9d624a6caa4798a66d8583aae59580
I20260927 15:02:16.063467 2037 ts_tablet_manager.cc:1939] T 8a9d624a6caa4798a66d8583aae59580 P abb3fd20a1d34ec2a598d73df66a3ba3: Deleting consensus metadata
I20260927 15:02:16.063642 2036 ts_tablet_manager.cc:1939] T 8a9d624a6caa4798a66d8583aae59580 P b0f0488caa644335ac54cf4571e46338: Deleting consensus metadata
I20260927 15:02:16.064438 1493 catalog_manager.cc:4841] TS abb3fd20a1d34ec2a598d73df66a3ba3 (127.0.98.3:36889): tablet 8a9d624a6caa4798a66d8583aae59580 (table test-table [id=e701210e8b8b49298202dc6ae2c1b8db]) successfully deleted
I20260927 15:02:16.064641 2038 ts_tablet_manager.cc:1905] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:16.067054 2038 ts_tablet_manager.cc:1918] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:16.067147 2038 log.cc:1196] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/8a9d624a6caa4798a66d8583aae59580
I20260927 15:02:16.067368 2038 ts_tablet_manager.cc:1939] T 8a9d624a6caa4798a66d8583aae59580 P 3f95705c47f34a668113e7d248c4405c: Deleting consensus metadata
I20260927 15:02:16.067987 1491 catalog_manager.cc:4841] TS b0f0488caa644335ac54cf4571e46338 (127.0.98.2:44407): tablet 8a9d624a6caa4798a66d8583aae59580 (table test-table [id=e701210e8b8b49298202dc6ae2c1b8db]) successfully deleted
I20260927 15:02:16.068032 1494 catalog_manager.cc:4841] TS 3f95705c47f34a668113e7d248c4405c (127.0.98.1:38937): tablet 8a9d624a6caa4798a66d8583aae59580 (table test-table [id=e701210e8b8b49298202dc6ae2c1b8db]) successfully deleted
Sep 27 15:02:16 dist-test-slave-ml7t krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:16.079660 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:16.069508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.0.98.1:38937)
0927 15:02:16.069641 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:16.069646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:16.069886 (+ 240us) server_negotiation.cc:407] Connection header received
0927 15:02:16.070022 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:16.070025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:16.070073 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:16.070151 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:16.070875 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.071606 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.072413 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.072761 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.072863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:16.076074 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:16.076084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:16.076099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:16.076124 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:16.077942 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:16.078339 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:16.078345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:16.078347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:16.078448 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:16.078686 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:16.078691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:16.078692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:16.078960 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:16.079153 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:16.079312 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 15:02:16.079503 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 15:02:16.080127 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45764
I20260927 15:02:16.089643 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:16.081868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.0.98.62:46823)
0927 15:02:16.081999 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:16.082003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:16.082018 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:02:16.082154 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:16.082156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:16.082197 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:16.082263 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:16.083053 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.083563 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.084647 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:16.084857 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:16.084925 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:16.086464 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:16.086468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:16.086470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:16.086493 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:16.088017 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:16.088444 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:16.088446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:16.088447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:16.088516 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:16.088781 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:16.088783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:16.088784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:16.088990 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:16.089119 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:16.089360 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:02:16.089458 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 15:02:16.090179 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38930
I20260927 15:02:16.094702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 1579
I20260927 15:02:16.101122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 1719
I20260927 15:02:16.106624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 1859
I20260927 15:02:16.112303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 1475
2026-09-27T15:02:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2891 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:16 dist-test-slave-ml7t krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:16 dist-test-slave-ml7t krb5kdc[2076](info): set up 2 sockets
Sep 27 15:02:16 dist-test-slave-ml7t krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260927 15:02:17.185539 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:17 dist-test-slave-ml7t krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:17.308702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37893
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:17.419067 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:17.419368 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:17.419610 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:17.423719 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:17.423826 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:02:17.423862 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:17.423904 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:17.423949 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:17.423983 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:17.429188 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:17.430228 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:17.431208 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:17.439410 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:17.439433 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:17.439422 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:17.439692 2089 server_base.cc:1034] running on GCE node
I20260927 15:02:17.440302 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:17.441447 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:17.442619 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790521337442572 us; error 62 us; skew 500 ppm
Sep 27 15:02:17 dist-test-slave-ml7t krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:17.445346 2089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:17.446354 2089 webserver.cc:460] Webserver started at http://127.0.98.62:46561/ using document root <none> and password file <none>
I20260927 15:02:17.446918 2089 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:17.446985 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:17.447217 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:17.449236 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "59e8b51a54c041b2a16428f00d4081ca"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "ae7862ce6446c0d8c4cb22caece557aa"
server_key_iv: "6b48927f696e86c655547b6304bb5498"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.449849 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "59e8b51a54c041b2a16428f00d4081ca"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "ae7862ce6446c0d8c4cb22caece557aa"
server_key_iv: "6b48927f696e86c655547b6304bb5498"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.453455 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:17.455835 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:17.457074 2089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:17.457198 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "59e8b51a54c041b2a16428f00d4081ca"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "ae7862ce6446c0d8c4cb22caece557aa"
server_key_iv: "6b48927f696e86c655547b6304bb5498"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.457297 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:17.484835 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:17.487671 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:17.487860 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:17.495441 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37893
I20260927 15:02:17.495589 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37893 every 8 connection(s)
I20260927 15:02:17.496407 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:17.499310 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:17.504773 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Bootstrap starting.
I20260927 15:02:17.505143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 2089
I20260927 15:02:17.505271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:17.505575 392 external_mini_cluster.cc:1428] Setting key 845248e44e6ceaf2eee108e0c6cf7d80
I20260927 15:02:17.507714 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:17.508782 2167 log.cc:824] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:17.510954 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: No bootstrap required, opened a new log
I20260927 15:02:17.514550 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8b51a54c041b2a16428f00d4081ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37893 } }
I20260927 15:02:17.514824 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:17.514918 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e8b51a54c041b2a16428f00d4081ca, State: Initialized, Role: FOLLOWER
Sep 27 15:02:17 dist-test-slave-ml7t krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:17.515471 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8b51a54c041b2a16428f00d4081ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37893 } }
I20260927 15:02:17.515658 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:17.515766 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:17.515885 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:17.517133 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8b51a54c041b2a16428f00d4081ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37893 } }
I20260927 15:02:17.517493 2167 leader_election.cc:302] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e8b51a54c041b2a16428f00d4081ca; no voters:
I20260927 15:02:17.517877 2167 leader_election.cc:288] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:17.517972 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:17.518152 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [term 1 LEADER]: Becoming Leader. State: Replica: 59e8b51a54c041b2a16428f00d4081ca, State: Running, Role: LEADER
I20260927 15:02:17.518471 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8b51a54c041b2a16428f00d4081ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37893 } }
I20260927 15:02:17.519237 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:17.520382 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e8b51a54c041b2a16428f00d4081ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8b51a54c041b2a16428f00d4081ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37893 } } }
I20260927 15:02:17.520519 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:17.520826 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e8b51a54c041b2a16428f00d4081ca. Latest consensus state: current_term: 1 leader_uuid: "59e8b51a54c041b2a16428f00d4081ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e8b51a54c041b2a16428f00d4081ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37893 } } }
I20260927 15:02:17.520889 2181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:17.520910 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:17.521173 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:17.506992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.0.98.62:37893)
0927 15:02:17.507548 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:17.507564 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:17.507600 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:02:17.508745 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:17.508773 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:17.509244 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:17.509591 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:17.510434 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.511461 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.512333 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.512818 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.512983 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:17.515707 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:17.515713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:17.515730 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:17.515785 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:17.517735 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:17.518284 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:17.518292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:17.518299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:17.518436 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:17.518759 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:17.518762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:17.518764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:17.519159 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:17.519376 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:17.519739 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 15:02:17.520052 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":185,"threads_started":1}
I20260927 15:02:17.525539 2181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:17.531343 2181 catalog_manager.cc:1347] Generated new cluster ID: 6bb1d9be8455491f932688de0f38fa0b
I20260927 15:02:17.531424 2181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:17.542865 2181 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:17.543807 2181 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:17.550940 2181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Generated new TSK 0
I20260927 15:02:17.551616 2181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:17.585189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37893
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:17.699841 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:17.700137 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:17.700212 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:17.704383 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:17.704475 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:17.704542 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:02:17.704586 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:17.704644 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:17.709815 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:17.710928 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:17.711944 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:17.721388 2193 server_base.cc:1034] running on GCE node
W20260927 15:02:17.721377 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:17.721355 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:17.721355 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:17.722009 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:17.722597 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:17.723799 2193 hybrid_clock.cc:648] HybridClock initialized: now 1790521337723784 us; error 28 us; skew 500 ppm
Sep 27 15:02:17 dist-test-slave-ml7t krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:17.726733 2193 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:17.727620 2193 webserver.cc:460] Webserver started at http://127.0.98.1:33623/ using document root <none> and password file <none>
I20260927 15:02:17.728173 2193 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:17.728266 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:17.728497 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:17.730562 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "daa4ad9e395a4519bacb3ca44e1e1c83"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "4310cec65fb75a52ae0eb555f72aaad2"
server_key_iv: "e7e5b7e1d226db95278f82dda9700b80"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.731099 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "daa4ad9e395a4519bacb3ca44e1e1c83"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "4310cec65fb75a52ae0eb555f72aaad2"
server_key_iv: "e7e5b7e1d226db95278f82dda9700b80"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.734782 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:17.737174 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:17.738372 2193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:17.738525 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "daa4ad9e395a4519bacb3ca44e1e1c83"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "4310cec65fb75a52ae0eb555f72aaad2"
server_key_iv: "e7e5b7e1d226db95278f82dda9700b80"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.738632 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:17.772056 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:17.775449 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:17.775663 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:17.777109 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:17.777181 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:17.777253 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:17.777297 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:17.788012 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36623
Sep 27 15:02:17 dist-test-slave-ml7t krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:17.788096 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36623 every 8 connection(s)
I20260927 15:02:17.789018 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:17.791877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 2193
I20260927 15:02:17.791980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:17.792306 392 external_mini_cluster.cc:1428] Setting key 693ae4ec759d707884249f7fdd0080f8
I20260927 15:02:17.792486 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:17.778121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56027 (local address 127.0.98.62:37893)
0927 15:02:17.778250 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:17.778254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:17.779278 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 15:02:17.780394 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:17.780399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:17.780455 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:17.780531 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:17.782413 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.783308 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.785055 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.785436 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.785569 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:17.788655 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:17.788662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:17.788668 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:17.788700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:17.790561 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:17.791049 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:17.791052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:17.791053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:17.791131 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:17.791411 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:17.791413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:17.791414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:17.791581 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:17.791655 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:17.792184 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 15:02:17.792314 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:02:17.793367 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:17.778657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37893 (local address 127.0.98.1:56027)
0927 15:02:17.779123 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:17.779164 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:17.780143 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:17.780970 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:17.780980 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:17.781424 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:17.782236 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:17.782256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.783459 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.783465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:17.784943 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:17.784953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.785662 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.785666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:17.785879 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:17.786623 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:17.786646 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:17.788490 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:17.790682 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:17.790686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:17.790697 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:17.790953 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:17.791211 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:17.791213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:17.791214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:17.791335 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:17.791667 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:17.791671 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:17.791880 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 15:02:17.792100 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 15:02:17.792376 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":150,"threads_started":1}
I20260927 15:02:17.794661 2324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37893
I20260927 15:02:17.794808 2324 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:17.795280 2324 heartbeater.cc:511] Master 127.0.98.62:37893 requested a full tablet report, sending...
I20260927 15:02:17.796964 2121 ts_manager.cc:194] Registered new tserver with Master: daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623)
I20260927 15:02:17.798614 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56027
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:17.806926 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:17.799281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44587 (local address 127.0.98.62:37893)
0927 15:02:17.799416 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:17.799422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:17.799440 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:02:17.799493 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:17.799497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:17.799549 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:17.799645 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:17.800145 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.801004 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.801823 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.802186 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.802319 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:17.803535 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:17.803538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:17.803540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:17.803561 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:17.804890 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:17.805496 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:17.805499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:17.805500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:17.805574 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:17.806059 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:17.806066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:17.806071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:17.806317 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:17.806402 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:17.806715 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 15:02:17.806808 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:02:17.807147 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:17.796911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37893 (local address 127.0.98.1:44587)
0927 15:02:17.797166 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:17.797177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:17.797245 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:17.799673 (+ 2428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:17.799677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:17.799897 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:17.800009 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:17.800017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.801138 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.801140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:17.801681 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:17.801688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:17.802358 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:17.802361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:17.802438 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:17.803050 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:17.803071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:17.803397 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:17.805061 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:17.805064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:17.805066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:17.805396 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:17.805723 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:17.805728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:17.805730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:17.805874 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:17.806420 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:17.806423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:17.806539 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 15:02:17.806662 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:02:17.806840 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:17.821050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37893
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:17.932847 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:17.933123 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:17.933172 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:17.937386 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:17.937441 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:17.937481 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:02:17.937496 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:17.937528 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:17.942718 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:17.943665 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:17.944607 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:17.954097 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:17.954165 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:17.954170 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:17.954279 2333 server_base.cc:1034] running on GCE node
I20260927 15:02:17.954779 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:17.955442 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:17.956630 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790521337956612 us; error 28 us; skew 500 ppm
Sep 27 15:02:17 dist-test-slave-ml7t krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:17.959327 2333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:17.960225 2333 webserver.cc:460] Webserver started at http://127.0.98.2:39883/ using document root <none> and password file <none>
I20260927 15:02:17.960851 2333 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:17.960950 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:17.961163 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:17.963181 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "d31c2f5e8db545bdac6077d5df4551cc"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "6092cf28d096022d94e0970d9219770b"
server_key_iv: "a676e2718d425dd68f6d01900f651d52"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.963723 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d31c2f5e8db545bdac6077d5df4551cc"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "6092cf28d096022d94e0970d9219770b"
server_key_iv: "a676e2718d425dd68f6d01900f651d52"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.967283 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:02:17.969542 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:17.970762 2333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:17.970887 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "d31c2f5e8db545bdac6077d5df4551cc"
format_stamp: "Formatted at 2026-09-27 15:02:17 on dist-test-slave-ml7t"
server_key: "6092cf28d096022d94e0970d9219770b"
server_key_iv: "a676e2718d425dd68f6d01900f651d52"
server_key_version: "encryptionkey@0"
I20260927 15:02:17.970994 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:18.013473 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:18.016814 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:18.017088 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:18.018731 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:18.018814 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:18.018893 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:18.018934 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:02:18 dist-test-slave-ml7t krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:18.030551 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41643
I20260927 15:02:18.030627 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41643 every 8 connection(s)
I20260927 15:02:18.031534 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:18.033723 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.019605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49965 (local address 127.0.98.62:37893)
0927 15:02:18.019751 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.019756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.020840 (+ 1084us) server_negotiation.cc:407] Connection header received
0927 15:02:18.021958 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.021963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.022022 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.022135 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:18.023825 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.024710 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.026424 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.026650 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.026767 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.029867 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:18.029870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:18.029872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:18.029895 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:18.031562 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.032166 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.032170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.032171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.032261 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.032630 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.032632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.032633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.032855 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:18.032954 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.033513 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.033608 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:02:18.034559 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.020154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37893 (local address 127.0.98.2:49965)
0927 15:02:18.020700 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.020749 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.021739 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.022548 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.022560 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.023042 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:18.023675 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.023690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.024845 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.024849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.026305 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.026316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.026809 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.026813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.026982 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.027576 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:18.027596 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:18.029671 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:18.031683 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.031689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.031702 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.032036 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.032357 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.032360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.032362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.032537 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.033028 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:18.033034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:18.033250 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.033397 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.033673 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260927 15:02:18.035750 2464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37893
I20260927 15:02:18.035950 2464 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:18.036489 2464 heartbeater.cc:511] Master 127.0.98.62:37893 requested a full tablet report, sending...
I20260927 15:02:18.037485 2121 ts_manager.cc:194] Registered new tserver with Master: d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643)
I20260927 15:02:18.037555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 2333
I20260927 15:02:18.037626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:18.037909 392 external_mini_cluster.cc:1428] Setting key 4ab8e502fabc2807becabd27b8335d21
I20260927 15:02:18.038277 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49965
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:18.048861 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.038116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34993 (local address 127.0.98.62:37893)
0927 15:02:18.038221 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.038226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.038483 (+ 257us) server_negotiation.cc:407] Connection header received
0927 15:02:18.038604 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.038607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.038656 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.038718 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:18.039265 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.039899 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.042710 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.043189 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.043328 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.044686 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:18.044693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:18.044698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:18.044738 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:18.046858 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.047371 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.047376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.047379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.047485 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.047783 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.047789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.047791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.048119 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:18.048228 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.048565 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.048685 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:02:18.048878 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.038038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37893 (local address 127.0.98.2:34993)
0927 15:02:18.038394 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.038408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.038522 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.038759 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.038762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.039015 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:18.039140 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.039146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.041913 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.041916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.042579 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.042587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.043435 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.043439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.043520 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.044149 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:18.044161 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:18.044506 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:18.047003 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.047005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.047007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.047261 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.047595 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.047597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.047599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.047681 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.048246 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:18.048248 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:18.048334 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.048505 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.048674 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:18.066857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37893
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:18.177163 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:18.177443 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:18.177493 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:18.181535 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:18.181602 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:18.181643 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:02:18.181656 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:18.181746 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:18.187085 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:18.188179 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:18.189203 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:18.198611 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:18.198638 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:18.198647 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:18.198702 2473 server_base.cc:1034] running on GCE node
I20260927 15:02:18.199211 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:18.199846 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:18.201109 2473 hybrid_clock.cc:648] HybridClock initialized: now 1790521338201090 us; error 41 us; skew 500 ppm
Sep 27 15:02:18 dist-test-slave-ml7t krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:18.203884 2473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:18.204684 2473 webserver.cc:460] Webserver started at http://127.0.98.3:34617/ using document root <none> and password file <none>
I20260927 15:02:18.205259 2473 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:18.205348 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:18.205570 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:18.207683 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2"
format_stamp: "Formatted at 2026-09-27 15:02:18 on dist-test-slave-ml7t"
server_key: "73406c5fe6eacbe9e15b7dbd67c1e6f1"
server_key_iv: "fbc0934797eef3df4cd7fd5a0f66c117"
server_key_version: "encryptionkey@0"
I20260927 15:02:18.208252 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2"
format_stamp: "Formatted at 2026-09-27 15:02:18 on dist-test-slave-ml7t"
server_key: "73406c5fe6eacbe9e15b7dbd67c1e6f1"
server_key_iv: "fbc0934797eef3df4cd7fd5a0f66c117"
server_key_version: "encryptionkey@0"
I20260927 15:02:18.211848 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:02:18.214160 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:18.215286 2473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:18.215430 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2"
format_stamp: "Formatted at 2026-09-27 15:02:18 on dist-test-slave-ml7t"
server_key: "73406c5fe6eacbe9e15b7dbd67c1e6f1"
server_key_iv: "fbc0934797eef3df4cd7fd5a0f66c117"
server_key_version: "encryptionkey@0"
I20260927 15:02:18.215533 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:18.232626 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:18.235973 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:18.236168 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:18.237649 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:18.237717 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:18.237779 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:18.237864 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:02:18 dist-test-slave-ml7t krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:18.248603 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40327
I20260927 15:02:18.249850 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:18.250022 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40327 every 8 connection(s)
I20260927 15:02:18.252876 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.238596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53773 (local address 127.0.98.62:37893)
0927 15:02:18.238761 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.238768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.239541 (+ 773us) server_negotiation.cc:407] Connection header received
0927 15:02:18.240617 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.240623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.240683 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.240740 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:18.242318 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.243298 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.244549 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.244882 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.245011 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.248149 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:18.248154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:18.248158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:18.248189 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:18.250338 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.251067 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.251070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.251072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.251160 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.251723 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.251727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.251728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.251966 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:18.252077 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.252567 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.252684 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:18.252985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 2473
I20260927 15:02:18.253111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:18.253373 392 external_mini_cluster.cc:1428] Setting key 596a4675ccc0e1c3cb7157974debccdb
I20260927 15:02:18.253734 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.238947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37893 (local address 127.0.98.3:53773)
0927 15:02:18.239374 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.239422 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.240340 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.241102 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.241108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.241517 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:18.242114 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.242128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.243411 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.243414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.244428 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.244440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.245062 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.245066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.245242 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.246424 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:18.246448 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:18.247979 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:18.250538 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.250545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.250560 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.250938 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.251290 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.251294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.251296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.251589 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.252086 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:18.252095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:18.252330 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.252523 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.252831 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":127,"threads_started":1}
I20260927 15:02:18.255021 2604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37893
I20260927 15:02:18.255148 2604 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:18.255803 2604 heartbeater.cc:511] Master 127.0.98.62:37893 requested a full tablet report, sending...
I20260927 15:02:18.256812 2120 ts_manager.cc:194] Registered new tserver with Master: 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327)
I20260927 15:02:18.257642 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53773
I20260927 15:02:18.264088 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.257136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37893 (local address 127.0.98.3:46741)
0927 15:02:18.257458 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.257476 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.257553 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.257799 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.257802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.258038 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:18.258149 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.258155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.258919 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.258922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.259682 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.259690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.260150 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.260154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.260238 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.260736 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:18.260746 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:18.261027 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:18.262885 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.262888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.262889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.263078 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.263317 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.263319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.263321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.263391 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.263700 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:18.263702 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:18.263770 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.263849 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.263947 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
I20260927 15:02:18.264213 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.257203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46741 (local address 127.0.98.62:37893)
0927 15:02:18.257330 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.257334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.257550 (+ 216us) server_negotiation.cc:407] Connection header received
0927 15:02:18.257616 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.257620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.257669 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.257761 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:18.258259 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.258790 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.259783 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.260044 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.260127 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.261143 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:18.261146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:18.261148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:18.261169 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:18.262792 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.263166 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.263168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.263169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.263241 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.263466 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.263468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.263470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.263624 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:18.263702 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.263959 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.264070 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:18.267354 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:02:18 dist-test-slave-ml7t krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:02:18 dist-test-slave-ml7t krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:18.293430 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.285172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.0.98.62:37893)
0927 15:02:18.285290 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.285293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.285306 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:02:18.285345 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.285347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.285383 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.285459 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:18.286247 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.286779 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.287615 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.287832 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.287975 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.290486 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:18.290489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:18.290491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:18.290516 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:18.292021 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.292385 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.292389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.292391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.292465 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.292687 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.292689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.292690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.292855 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:18.293000 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.293219 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.293310 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:02:18.296471 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:18.299443 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:18.313138 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.307112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.0.1:55260)
0927 15:02:18.307882 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.307933 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.308066 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.308812 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.308822 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.308849 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.309013 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.309021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.310173 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.310177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.311420 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.311430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.312469 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.312472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.312597 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.312642 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.312757 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.312985 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":75,"spinlock_wait_cycles":6016,"thread_start_us":100,"threads_started":1}
I20260927 15:02:18.313261 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.307204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.0.98.3:40327)
0927 15:02:18.308501 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.308512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.308534 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:02:18.308603 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.308611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.308772 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.309013 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.309239 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.310506 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.311884 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.312671 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.312778 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.312921 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.313025 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.313110 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1193,"thread_start_us":115,"threads_started":1}
I20260927 15:02:18.313268 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.307086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40327 (local address 127.0.0.1:48118)
0927 15:02:18.307882 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.307933 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.308066 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.308981 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.308984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.309000 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.309112 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.309118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.310624 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.310627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.311764 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.311772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.312788 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.312791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.312880 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.312903 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.312999 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.313081 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"spinlock_wait_cycles":3328,"thread_start_us":82,"threads_started":1}
I20260927 15:02:18.313949 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.307504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.0.98.1:36623)
0927 15:02:18.307772 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.307779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.308218 (+ 439us) server_negotiation.cc:407] Connection header received
0927 15:02:18.308299 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.308306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.308518 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.308660 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.309140 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.310051 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.311559 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.312340 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.312447 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.312578 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.313699 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.313824 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260927 15:02:18.316069 2538 tablet_service.cc:1467] Processing CreateTablet for tablet f94c811921bc482a9ec34f1214f370cd (DEFAULT_TABLE table=test-table [id=5896c39e12344b3eb92a0d48abb52e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:18.316094 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f94c811921bc482a9ec34f1214f370cd (DEFAULT_TABLE table=test-table [id=5896c39e12344b3eb92a0d48abb52e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:18.317062 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94c811921bc482a9ec34f1214f370cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.317073 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94c811921bc482a9ec34f1214f370cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.317305 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.308237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.0.98.2:41643)
0927 15:02:18.308660 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.308671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.308693 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:02:18.308766 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.308771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.308968 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.309111 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.309377 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.310513 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.313126 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.314185 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.316607 (+ 2422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.316761 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.317081 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.317174 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":116,"threads_started":1}
I20260927 15:02:18.317322 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.307828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41643 (local address 127.0.0.1:39934)
0927 15:02:18.308046 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.308059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.308166 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.309169 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.309172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.309191 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.309267 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.309272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.311884 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.311887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.313005 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.313013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.316821 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.316824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.316913 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.316936 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.317030 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.317197 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":48,"thread_start_us":96,"threads_started":1}
I20260927 15:02:18.320712 2398 tablet_service.cc:1467] Processing CreateTablet for tablet f94c811921bc482a9ec34f1214f370cd (DEFAULT_TABLE table=test-table [id=5896c39e12344b3eb92a0d48abb52e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:18.321851 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94c811921bc482a9ec34f1214f370cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.323696 2627 tablet_bootstrap.cc:492] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Bootstrap starting.
I20260927 15:02:18.325398 2628 tablet_bootstrap.cc:492] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Bootstrap starting.
I20260927 15:02:18.325963 2627 tablet_bootstrap.cc:654] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.326802 2627 log.cc:824] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:18.327492 2628 tablet_bootstrap.cc:654] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.328290 2628 log.cc:824] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:18.330158 2628 tablet_bootstrap.cc:492] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: No bootstrap required, opened a new log
I20260927 15:02:18.330222 2627 tablet_bootstrap.cc:492] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: No bootstrap required, opened a new log
I20260927 15:02:18.330369 2628 ts_tablet_manager.cc:1397] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:02:18.330399 2627 ts_tablet_manager.cc:1397] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 15:02:18.333652 2628 raft_consensus.cc:348] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.333962 2628 raft_consensus.cc:374] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.334034 2628 raft_consensus.cc:729] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f56f3ecdc42148a9ca8bd7ae5f6e2, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.334034 2627 raft_consensus.cc:348] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.334283 2627 raft_consensus.cc:374] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.334340 2627 raft_consensus.cc:729] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa4ad9e395a4519bacb3ca44e1e1c83, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.334475 2628 consensus_queue.cc:260] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.334780 2627 consensus_queue.cc:260] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.335176 2604 heartbeater.cc:503] Master 127.0.98.62:37893 was elected leader, sending a full tablet report...
I20260927 15:02:18.335469 2628 ts_tablet_manager.cc:1428] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:02:18.335719 2627 ts_tablet_manager.cc:1428] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:02:18.336011 2324 heartbeater.cc:503] Master 127.0.98.62:37893 was elected leader, sending a full tablet report...
I20260927 15:02:18.336170 2631 tablet_bootstrap.cc:492] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Bootstrap starting.
I20260927 15:02:18.338397 2631 tablet_bootstrap.cc:654] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.339282 2631 log.cc:824] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:18.341331 2631 tablet_bootstrap.cc:492] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: No bootstrap required, opened a new log
I20260927 15:02:18.341539 2631 ts_tablet_manager.cc:1397] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:02:18.344533 2631 raft_consensus.cc:348] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.344725 2631 raft_consensus.cc:374] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.344780 2631 raft_consensus.cc:729] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31c2f5e8db545bdac6077d5df4551cc, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.345184 2631 consensus_queue.cc:260] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.345821 2464 heartbeater.cc:503] Master 127.0.98.62:37893 was elected leader, sending a full tablet report...
I20260927 15:02:18.346133 2631 ts_tablet_manager.cc:1428] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 15:02:18.500944 2605 tablet.cc:2369] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:18.532889 2465 tablet.cc:2369] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:18.540818 2325 tablet.cc:2369] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:18.573103 2635 raft_consensus.cc:482] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:18.573348 2635 raft_consensus.cc:504] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.574594 2635 leader_election.cc:288] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327), daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623)
I20260927 15:02:18.580480 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.575140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35557 (local address 127.0.98.1:36623)
0927 15:02:18.575270 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.575274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.575486 (+ 212us) server_negotiation.cc:407] Connection header received
0927 15:02:18.575731 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.575735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.575791 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.575887 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.576374 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.577167 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.579143 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.579783 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.579911 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.579982 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.580271 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.580344 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:02:18.580518 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.575080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51125 (local address 127.0.98.3:40327)
0927 15:02:18.575224 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.575229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.575537 (+ 308us) server_negotiation.cc:407] Connection header received
0927 15:02:18.575731 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.575735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.575791 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.575895 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.576400 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.577203 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.579143 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.579783 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.579911 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.579982 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.580271 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.580344 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:02:18.580546 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.574924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40327 (local address 127.0.98.2:51125)
0927 15:02:18.575442 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.575457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.575562 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.575946 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.575949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.575981 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.576244 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.576271 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.577329 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.577333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.578989 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.579000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.579948 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.579952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.580109 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.580150 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.580238 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.580358 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":436,"spinlock_wait_cycles":8960,"thread_start_us":62,"threads_started":1}
I20260927 15:02:18.580544 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.574922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.98.2:35557)
0927 15:02:18.575375 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.575393 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.575562 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.575946 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.575949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.575981 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.576240 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.576248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.577329 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.577333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.578989 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.579000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.579948 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.579952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.580106 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.580158 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.580238 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.580361 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":334,"spinlock_wait_cycles":38400,"thread_start_us":110,"threads_started":1}
I20260927 15:02:18.581319 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94c811921bc482a9ec34f1214f370cd" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" is_pre_election: true
I20260927 15:02:18.581552 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94c811921bc482a9ec34f1214f370cd" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" is_pre_election: true
I20260927 15:02:18.581636 2545 raft_consensus.cc:2454] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 0.
I20260927 15:02:18.581899 2278 raft_consensus.cc:2454] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 0.
I20260927 15:02:18.582204 2352 leader_election.cc:302] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, d31c2f5e8db545bdac6077d5df4551cc; no voters:
I20260927 15:02:18.582522 2635 raft_consensus.cc:2781] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:18.582597 2635 raft_consensus.cc:482] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:18.582628 2635 raft_consensus.cc:3037] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.583631 2635 raft_consensus.cc:504] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.584002 2635 leader_election.cc:288] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 election: Requested vote from peers 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327), daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623)
I20260927 15:02:18.584318 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94c811921bc482a9ec34f1214f370cd" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2"
I20260927 15:02:18.584439 2545 raft_consensus.cc:3037] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.585762 2545 raft_consensus.cc:2454] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 1.
I20260927 15:02:18.585914 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94c811921bc482a9ec34f1214f370cd" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83"
I20260927 15:02:18.586057 2278 raft_consensus.cc:3037] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.586189 2352 leader_election.cc:302] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, d31c2f5e8db545bdac6077d5df4551cc; no voters:
I20260927 15:02:18.586380 2635 raft_consensus.cc:2781] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:18.586577 2635 raft_consensus.cc:686] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 1 LEADER]: Becoming Leader. State: Replica: d31c2f5e8db545bdac6077d5df4551cc, State: Running, Role: LEADER
I20260927 15:02:18.587100 2635 consensus_queue.cc:237] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } }
I20260927 15:02:18.587275 2278 raft_consensus.cc:2454] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 1.
I20260927 15:02:18.590260 2121 catalog_manager.cc:5495] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc reported cstate change: term changed from 0 to 1, leader changed from <none> to d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d31c2f5e8db545bdac6077d5df4551cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } health_report { overall_health: HEALTHY } } }
Sep 27 15:02:18 dist-test-slave-ml7t krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:18.631819 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.623565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.0.98.62:37893)
0927 15:02:18.623724 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.623729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.623832 (+ 103us) server_negotiation.cc:407] Connection header received
0927 15:02:18.624005 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.624007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.624045 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.624115 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:18.624761 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.625290 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.626257 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.626450 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.626528 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.628813 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:18.628816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:18.628818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:18.628836 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:18.630146 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.630591 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.630598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.630600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.630672 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:18.630935 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:18.630940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:18.630942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:18.631112 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:18.631217 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.631496 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.631653 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":77}
I20260927 15:02:18.631819 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.623425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37893 (local address 127.0.0.1:56624)
0927 15:02:18.623737 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.623751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.623907 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.624160 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.624162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.624558 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:18.624653 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.624659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.625397 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.625400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.626149 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.626158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.626545 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.626547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.626605 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.627222 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:18.627233 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:18.628675 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:18.630251 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.630262 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.630269 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.630494 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.630763 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:18.630768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:18.630770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:18.630848 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:18.631224 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:18.631233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:18.631337 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.631452 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.631606 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260927 15:02:18.635705 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56624:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:02:18.636472 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:18.641607 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 4836daff3b004bdba7f3a690699bc94f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8057ea8993bd4e5080e248194922b9a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:18.641607 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 4836daff3b004bdba7f3a690699bc94f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8057ea8993bd4e5080e248194922b9a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:18.641709 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 4836daff3b004bdba7f3a690699bc94f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8057ea8993bd4e5080e248194922b9a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:18.642391 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4836daff3b004bdba7f3a690699bc94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.642391 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4836daff3b004bdba7f3a690699bc94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.642521 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4836daff3b004bdba7f3a690699bc94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.645331 2628 tablet_bootstrap.cc:492] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Bootstrap starting.
I20260927 15:02:18.645934 2631 tablet_bootstrap.cc:492] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: Bootstrap starting.
I20260927 15:02:18.646574 2628 tablet_bootstrap.cc:654] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.647393 2631 tablet_bootstrap.cc:654] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.648083 2627 tablet_bootstrap.cc:492] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: Bootstrap starting.
I20260927 15:02:18.648398 2628 tablet_bootstrap.cc:492] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: No bootstrap required, opened a new log
I20260927 15:02:18.648478 2628 ts_tablet_manager.cc:1397] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:18.648989 2628 raft_consensus.cc:348] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.649114 2627 tablet_bootstrap.cc:654] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.649137 2628 raft_consensus.cc:374] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.649173 2628 raft_consensus.cc:729] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f56f3ecdc42148a9ca8bd7ae5f6e2, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.649329 2628 consensus_queue.cc:260] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.649631 2632 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.649772 2632 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:18.649644 2628 ts_tablet_manager.cc:1428] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:02:18.650475 2631 tablet_bootstrap.cc:492] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: No bootstrap required, opened a new log
I20260927 15:02:18.650579 2631 ts_tablet_manager.cc:1397] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:02:18.651091 2631 raft_consensus.cc:348] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.651232 2631 raft_consensus.cc:374] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.651288 2631 raft_consensus.cc:729] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31c2f5e8db545bdac6077d5df4551cc, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.651469 2631 consensus_queue.cc:260] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.651757 2635 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.651901 2635 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:18.651773 2631 ts_tablet_manager.cc:1428] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:02:18.652544 2627 tablet_bootstrap.cc:492] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: No bootstrap required, opened a new log
I20260927 15:02:18.652652 2627 ts_tablet_manager.cc:1397] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:02:18.653206 2627 raft_consensus.cc:348] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.653331 2627 raft_consensus.cc:374] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.653419 2627 raft_consensus.cc:729] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa4ad9e395a4519bacb3ca44e1e1c83, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.653558 2627 consensus_queue.cc:260] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.653913 2627 ts_tablet_manager.cc:1428] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 15:02:18.653887 2633 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.654030 2633 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:18.658869 2633 raft_consensus.cc:482] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:18.658959 2633 raft_consensus.cc:504] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.660164 2633 leader_election.cc:288] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643), 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327)
I20260927 15:02:18.665130 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.660500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35639 (local address 127.0.98.2:41643)
0927 15:02:18.660598 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.660603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.660883 (+ 280us) server_negotiation.cc:407] Connection header received
0927 15:02:18.661001 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.661004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.661047 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.661116 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.661517 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.662277 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.663732 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.664369 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.664491 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.664558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.664912 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.665019 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:02:18.665141 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.660448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44459 (local address 127.0.98.3:40327)
0927 15:02:18.660566 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.660570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.660848 (+ 278us) server_negotiation.cc:407] Connection header received
0927 15:02:18.660986 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.660990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.661039 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.661117 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.661913 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.662562 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.663863 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.664430 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.664531 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.664583 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.664944 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.665033 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:02:18.665158 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.660389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41643 (local address 127.0.98.1:35639)
0927 15:02:18.660818 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.660830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.660904 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.661141 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.661144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.661185 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.661410 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.661418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.662395 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.662397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.663612 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.663622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.664560 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.664567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.664727 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.664794 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.664881 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.664970 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":81,"spinlock_wait_cycles":48256,"thread_start_us":129,"threads_started":1}
I20260927 15:02:18.665616 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.660389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40327 (local address 127.0.98.1:44459)
0927 15:02:18.660744 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.660759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.660869 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.661141 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.661144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.661170 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.661777 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.661804 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.662654 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.662656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.663753 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.663759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.664571 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.664574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.664722 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.664797 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.665360 (+ 563us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.665444 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":95,"spinlock_wait_cycles":1664,"thread_start_us":159,"threads_started":1}
I20260927 15:02:18.666086 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4836daff3b004bdba7f3a690699bc94f" candidate_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" is_pre_election: true
I20260927 15:02:18.666239 2545 raft_consensus.cc:2454] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daa4ad9e395a4519bacb3ca44e1e1c83 in term 0.
I20260927 15:02:18.666605 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4836daff3b004bdba7f3a690699bc94f" candidate_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31c2f5e8db545bdac6077d5df4551cc" is_pre_election: true
I20260927 15:02:18.666623 2212 leader_election.cc:302] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, daa4ad9e395a4519bacb3ca44e1e1c83; no voters:
I20260927 15:02:18.666815 2418 raft_consensus.cc:2454] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daa4ad9e395a4519bacb3ca44e1e1c83 in term 0.
I20260927 15:02:18.666913 2633 raft_consensus.cc:2781] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:18.666966 2633 raft_consensus.cc:482] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:18.667011 2633 raft_consensus.cc:3037] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.668004 2633 raft_consensus.cc:504] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.668294 2633 leader_election.cc:288] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [CANDIDATE]: Term 1 election: Requested vote from peers d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643), 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327)
I20260927 15:02:18.668618 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4836daff3b004bdba7f3a690699bc94f" candidate_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2"
I20260927 15:02:18.668768 2545 raft_consensus.cc:3037] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.669612 2545 raft_consensus.cc:2454] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa4ad9e395a4519bacb3ca44e1e1c83 in term 1.
I20260927 15:02:18.669893 2212 leader_election.cc:302] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, daa4ad9e395a4519bacb3ca44e1e1c83; no voters:
I20260927 15:02:18.669932 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4836daff3b004bdba7f3a690699bc94f" candidate_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31c2f5e8db545bdac6077d5df4551cc"
I20260927 15:02:18.670063 2418 raft_consensus.cc:3037] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.670104 2633 raft_consensus.cc:2781] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:18.670307 2633 raft_consensus.cc:686] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 LEADER]: Becoming Leader. State: Replica: daa4ad9e395a4519bacb3ca44e1e1c83, State: Running, Role: LEADER
I20260927 15:02:18.670653 2633 consensus_queue.cc:237] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.671218 2418 raft_consensus.cc:2454] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa4ad9e395a4519bacb3ca44e1e1c83 in term 1.
I20260927 15:02:18.672749 2651 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: TxnStatusTablet state changed. Reason: New leader daa4ad9e395a4519bacb3ca44e1e1c83. Latest consensus state: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.672852 2651 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:18.673125 2653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:02:18.673341 2653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:02:18.674006 2121 catalog_manager.cc:5495] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 reported cstate change: term changed from 0 to 1, leader changed from <none> to daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } health_report { overall_health: UNKNOWN } } }
I20260927 15:02:18.689949 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.686589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.0.98.1:36623)
0927 15:02:18.686714 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.686719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.686731 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:18.686792 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.686795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.686839 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.686947 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:18.687199 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.687704 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.688555 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.688774 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.688884 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.689180 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.689555 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.689677 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.689752 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.689817 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:02:18.689966 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.686338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.0.1:55274)
0927 15:02:18.686465 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.686483 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.686589 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.686966 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.686969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.686977 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:18.687082 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.687088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.687816 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.687822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.688431 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.688439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.688881 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.688885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.688984 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.689033 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.689655 (+ 622us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.689670 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.689741 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.689844 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 15:02:18.694909 2418 raft_consensus.cc:1261] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Refusing update from remote peer daa4ad9e395a4519bacb3ca44e1e1c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:18.694936 2545 raft_consensus.cc:1261] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Refusing update from remote peer daa4ad9e395a4519bacb3ca44e1e1c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:18.695555 2651 consensus_queue.cc:1035] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:18.695802 2633 consensus_queue.cc:1035] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:18.699234 2635 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: TxnStatusTablet state changed. Reason: New leader daa4ad9e395a4519bacb3ca44e1e1c83. Latest consensus state: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.699352 2635 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:18.700217 2632 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: TxnStatusTablet state changed. Reason: New leader daa4ad9e395a4519bacb3ca44e1e1c83. Latest consensus state: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.700320 2632 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:18.703269 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333975403287457792 to 7333975403197632512. Current Snapshot: MvccSnapshot[applied={T|T < 7333975403287457792}]
I20260927 15:02:18.703437 2651 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.703547 2651 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:18.703644 2651 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.703712 2651 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P daa4ad9e395a4519bacb3ca44e1e1c83: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:18.704190 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333975403287457792 to 7333975403197632512. Current Snapshot: MvccSnapshot[applied={T|T < 7333975403287457792}]
I20260927 15:02:18.704361 2635 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.704468 2635 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P d31c2f5e8db545bdac6077d5df4551cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:18.706136 2658 mvcc.cc:205] Tried to move back new op lower bound from 7333975403287457792 to 7333975403197632512. Current Snapshot: MvccSnapshot[applied={T|T < 7333975403287457792}]
I20260927 15:02:18.706259 2632 tablet_replica.cc:439] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } } }
I20260927 15:02:18.706357 2632 tablet_replica.cc:442] T 4836daff3b004bdba7f3a690699bc94f P 449f56f3ecdc42148a9ca8bd7ae5f6e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:18.718758 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.714872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.0.98.2:41643)
0927 15:02:18.714997 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.715001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.715036 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:02:18.715198 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.715202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.715262 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.715367 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:18.715571 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.716364 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.717160 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.717498 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.717598 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.717840 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.718323 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.718440 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.718578 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.718630 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:02:18.729257 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.725982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45605 (local address 127.0.98.1:36623)
0927 15:02:18.726120 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.726127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.726208 (+ 81us) server_negotiation.cc:407] Connection header received
0927 15:02:18.726357 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.726361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.726411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.726493 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:18.726751 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.727259 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.728188 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.728403 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.728509 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.728825 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.728905 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.728981 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.729073 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.729118 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:18.729283 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.725748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.98.2:45605)
0927 15:02:18.726115 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.726132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.726245 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.726503 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.726505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.726513 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:18.726649 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.726654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.727362 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.727369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.728067 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.728075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.728510 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.728513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.728625 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.728664 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.728995 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.729002 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.729062 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.729133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":125,"threads_started":1}
I20260927 15:02:18.735818 2545 raft_consensus.cc:1261] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Refusing update from remote peer d31c2f5e8db545bdac6077d5df4551cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:18.736142 2278 raft_consensus.cc:1261] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Refusing update from remote peer d31c2f5e8db545bdac6077d5df4551cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:18.736358 2638 consensus_queue.cc:1035] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:18.737062 2638 consensus_queue.cc:1035] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:18.761804 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.756460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41643 (local address 127.0.98.1:35721)
0927 15:02:18.758042 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.758055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.758180 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.758426 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.758429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.758437 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:18.758588 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.758598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.759600 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.759602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.760265 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.760273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.760963 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.760967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.761073 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.761126 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.761465 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.761474 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.761544 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.761620 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":108,"threads_started":1}
I20260927 15:02:18.761756 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.756513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35721 (local address 127.0.98.2:41643)
0927 15:02:18.756644 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.756648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.758131 (+ 1483us) server_negotiation.cc:407] Connection header received
0927 15:02:18.758268 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.758271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.758320 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.758404 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:18.758719 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.759492 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.760412 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.760817 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.760934 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.761289 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.761373 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.761451 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.761563 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.761612 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:02:18.787792 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:18.787984 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:18.790663 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Sending DeleteTablet for 3 replicas of tablet f94c811921bc482a9ec34f1214f370cd
I20260927 15:02:18.791401 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet f94c811921bc482a9ec34f1214f370cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55260
I20260927 15:02:18.791402 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet f94c811921bc482a9ec34f1214f370cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48118
I20260927 15:02:18.792130 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet f94c811921bc482a9ec34f1214f370cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39934
I20260927 15:02:18.792569 2679 tablet_replica.cc:330] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: stopping tablet replica
I20260927 15:02:18.792884 2679 raft_consensus.cc:2229] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:18.793298 2679 raft_consensus.cc:2258] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:18.793272 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:18.793715 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:18.795084 2679 ts_tablet_manager.cc:1905] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:18.798192 2679 ts_tablet_manager.cc:1918] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:18.798355 2679 log.cc:1196] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/f94c811921bc482a9ec34f1214f370cd
I20260927 15:02:18.798735 2679 ts_tablet_manager.cc:1939] T f94c811921bc482a9ec34f1214f370cd P d31c2f5e8db545bdac6077d5df4551cc: Deleting consensus metadata
I20260927 15:02:18.799675 2538 tablet_service.cc:1467] Processing CreateTablet for tablet c1365df26f8f4d33870853b1982ae0a8 (DEFAULT_TABLE table=test-table [id=14afc13a65f243cf8687b8e58b03ca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:18.799976 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1365df26f8f4d33870853b1982ae0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.802107 2106 catalog_manager.cc:4841] TS d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643): tablet f94c811921bc482a9ec34f1214f370cd (table test-table [id=5896c39e12344b3eb92a0d48abb52e6e]) successfully deleted
I20260927 15:02:18.802330 2258 tablet_service.cc:1467] Processing CreateTablet for tablet c1365df26f8f4d33870853b1982ae0a8 (DEFAULT_TABLE table=test-table [id=14afc13a65f243cf8687b8e58b03ca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:18.802597 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1365df26f8f4d33870853b1982ae0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.802644 2628 tablet_bootstrap.cc:492] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Bootstrap starting.
I20260927 15:02:18.803666 2628 tablet_bootstrap.cc:654] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.803942 2678 tablet_replica.cc:330] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: stopping tablet replica
I20260927 15:02:18.804280 2678 raft_consensus.cc:2229] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:18.804527 2678 raft_consensus.cc:2258] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:18.805174 2628 tablet_bootstrap.cc:492] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: No bootstrap required, opened a new log
I20260927 15:02:18.805143 2398 tablet_service.cc:1467] Processing CreateTablet for tablet c1365df26f8f4d33870853b1982ae0a8 (DEFAULT_TABLE table=test-table [id=14afc13a65f243cf8687b8e58b03ca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:18.805274 2628 ts_tablet_manager.cc:1397] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:18.805382 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1365df26f8f4d33870853b1982ae0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:18.805806 2628 raft_consensus.cc:348] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.805934 2628 raft_consensus.cc:374] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.805975 2628 raft_consensus.cc:729] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f56f3ecdc42148a9ca8bd7ae5f6e2, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.806108 2628 consensus_queue.cc:260] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.806408 2628 ts_tablet_manager.cc:1428] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:18.807044 2678 ts_tablet_manager.cc:1905] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:18.807726 2631 tablet_bootstrap.cc:492] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: Bootstrap starting.
I20260927 15:02:18.808640 2631 tablet_bootstrap.cc:654] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.809506 2627 tablet_bootstrap.cc:492] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: Bootstrap starting.
I20260927 15:02:18.810412 2627 tablet_bootstrap.cc:654] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:18.810917 2678 ts_tablet_manager.cc:1918] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:18.811050 2678 log.cc:1196] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/f94c811921bc482a9ec34f1214f370cd
I20260927 15:02:18.811398 2678 ts_tablet_manager.cc:1939] T f94c811921bc482a9ec34f1214f370cd P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting consensus metadata
I20260927 15:02:18.812608 2107 catalog_manager.cc:4841] TS daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623): tablet f94c811921bc482a9ec34f1214f370cd (table test-table [id=5896c39e12344b3eb92a0d48abb52e6e]) successfully deleted
I20260927 15:02:18.816998 2627 tablet_bootstrap.cc:492] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: No bootstrap required, opened a new log
I20260927 15:02:18.817103 2627 ts_tablet_manager.cc:1397] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 15:02:18.817242 2631 tablet_bootstrap.cc:492] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: No bootstrap required, opened a new log
I20260927 15:02:18.817314 2631 ts_tablet_manager.cc:1397] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 15:02:18.817526 2627 raft_consensus.cc:348] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.817654 2627 raft_consensus.cc:374] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.817688 2627 raft_consensus.cc:729] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa4ad9e395a4519bacb3ca44e1e1c83, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.817754 2631 raft_consensus.cc:348] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.817832 2627 consensus_queue.cc:260] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.817917 2631 raft_consensus.cc:374] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:18.817953 2631 raft_consensus.cc:729] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31c2f5e8db545bdac6077d5df4551cc, State: Initialized, Role: FOLLOWER
I20260927 15:02:18.818063 2631 consensus_queue.cc:260] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.818120 2627 ts_tablet_manager.cc:1428] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:18.818338 2631 ts_tablet_manager.cc:1428] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:02:18.820039 2677 tablet_replica.cc:330] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: stopping tablet replica
I20260927 15:02:18.820503 2677 raft_consensus.cc:2229] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:18.820806 2677 raft_consensus.cc:2258] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:18.827418 2677 ts_tablet_manager.cc:1905] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:18.829730 2677 ts_tablet_manager.cc:1918] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:18.829877 2677 log.cc:1196] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/f94c811921bc482a9ec34f1214f370cd
I20260927 15:02:18.830168 2677 ts_tablet_manager.cc:1939] T f94c811921bc482a9ec34f1214f370cd P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting consensus metadata
I20260927 15:02:18.831550 2108 catalog_manager.cc:4841] TS 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327): tablet f94c811921bc482a9ec34f1214f370cd (table test-table [id=5896c39e12344b3eb92a0d48abb52e6e]) successfully deleted
I20260927 15:02:18.857280 2632 raft_consensus.cc:482] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:18.857453 2632 raft_consensus.cc:504] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.858649 2632 leader_election.cc:288] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623), d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643)
I20260927 15:02:18.863605 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.858916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43833 (local address 127.0.98.1:36623)
0927 15:02:18.859045 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.859050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.859352 (+ 302us) server_negotiation.cc:407] Connection header received
0927 15:02:18.859562 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.859566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.859617 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.859667 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.860116 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.860885 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.862365 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.862924 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.863008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.863072 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.863401 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.863481 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:02:18.864202 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.858842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.98.3:43833)
0927 15:02:18.859268 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.859284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.859432 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.859812 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.859815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.859832 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.859984 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.859992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.861018 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.861021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.862238 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.862246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.863155 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.863158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.863256 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.863286 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.863995 (+ 709us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.864064 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":84,"threads_started":1}
I20260927 15:02:18.864738 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1365df26f8f4d33870853b1982ae0a8" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" is_pre_election: true
I20260927 15:02:18.864930 2278 raft_consensus.cc:2454] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 0.
I20260927 15:02:18.865065 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.859566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33277 (local address 127.0.98.2:41643)
0927 15:02:18.859684 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.859688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.859917 (+ 229us) server_negotiation.cc:407] Connection header received
0927 15:02:18.860098 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.860102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.860151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.860197 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:18.860655 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.861389 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.863755 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.864350 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.864443 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.864523 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.864871 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.864954 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:02:18.865458 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.859238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41643 (local address 127.0.98.3:33277)
0927 15:02:18.859837 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.859850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.859974 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.860376 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.860379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.860394 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:18.860533 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.860540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.861598 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.861601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.863621 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.863632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.864636 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.864640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.864729 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.864752 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.865236 (+ 484us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.865318 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":64,"threads_started":1}
I20260927 15:02:18.865500 2491 leader_election.cc:302] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, daa4ad9e395a4519bacb3ca44e1e1c83; no voters:
I20260927 15:02:18.865803 2632 raft_consensus.cc:2781] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:18.865857 2632 raft_consensus.cc:482] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:18.865913 2632 raft_consensus.cc:3037] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.865917 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1365df26f8f4d33870853b1982ae0a8" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31c2f5e8db545bdac6077d5df4551cc" is_pre_election: true
I20260927 15:02:18.866042 2418 raft_consensus.cc:2454] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 0.
I20260927 15:02:18.867195 2632 raft_consensus.cc:504] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.867571 2632 leader_election.cc:288] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 election: Requested vote from peers daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623), d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643)
I20260927 15:02:18.867885 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1365df26f8f4d33870853b1982ae0a8" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83"
I20260927 15:02:18.867962 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1365df26f8f4d33870853b1982ae0a8" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31c2f5e8db545bdac6077d5df4551cc"
I20260927 15:02:18.867998 2278 raft_consensus.cc:3037] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.868059 2418 raft_consensus.cc:3037] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:18.869169 2278 raft_consensus.cc:2454] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 1.
I20260927 15:02:18.869393 2418 raft_consensus.cc:2454] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 1.
I20260927 15:02:18.869580 2491 leader_election.cc:302] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, daa4ad9e395a4519bacb3ca44e1e1c83; no voters:
I20260927 15:02:18.869711 2632 raft_consensus.cc:2781] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:18.869933 2632 raft_consensus.cc:686] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 LEADER]: Becoming Leader. State: Replica: 449f56f3ecdc42148a9ca8bd7ae5f6e2, State: Running, Role: LEADER
I20260927 15:02:18.870190 2632 consensus_queue.cc:237] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:18.872562 2120 catalog_manager.cc:5495] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } health_report { overall_health: HEALTHY } } }
I20260927 15:02:18.888067 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.884461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.0.98.3:40327)
0927 15:02:18.884562 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.884568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.884616 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:02:18.884815 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.884818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.884872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.884998 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:18.885236 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.885817 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.886640 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.886870 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.886975 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.887236 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.887682 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.887806 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.887871 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.887924 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:02:18.899545 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.895214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57541 (local address 127.0.98.1:36623)
0927 15:02:18.895337 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.895341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.895501 (+ 160us) server_negotiation.cc:407] Connection header received
0927 15:02:18.895644 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.895646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.895722 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.895786 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:18.896088 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.896575 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.898492 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.898701 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.898783 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.899132 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.899200 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.899258 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.899395 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.899440 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:02:18.899670 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.895068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.98.3:57541)
0927 15:02:18.895407 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.895423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.895534 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.895802 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.895806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.895814 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:18.895949 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.895957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.896689 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.896692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.898362 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.898374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.898815 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.898819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.898932 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.898994 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.899294 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.899301 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.899379 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.899465 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260927 15:02:18.906383 2278 raft_consensus.cc:1261] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Refusing update from remote peer 449f56f3ecdc42148a9ca8bd7ae5f6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:18.906399 2418 raft_consensus.cc:1261] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Refusing update from remote peer 449f56f3ecdc42148a9ca8bd7ae5f6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:18.906942 2685 consensus_queue.cc:1035] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:18.907155 2632 consensus_queue.cc:1035] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:18.927886 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.923595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35399 (local address 127.0.98.3:40327)
0927 15:02:18.923698 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:18.923703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:18.923986 (+ 283us) server_negotiation.cc:407] Connection header received
0927 15:02:18.924035 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:18.924038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:18.924095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:18.924165 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:18.924536 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.925285 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.926637 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.926979 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.927092 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.927432 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.927522 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.927593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:18.927728 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 15:02:18.927777 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:02:18.927960 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:18.923561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40327 (local address 127.0.98.1:35399)
0927 15:02:18.923758 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:18.923775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:18.923886 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:18.924282 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:18.924285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:18.924294 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:18.924401 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.924408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.925561 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.925564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.926507 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:18.926518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:18.927177 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:18.927181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:18.927261 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:18.927294 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:18.927633 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:18.927641 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:02:18.927716 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:02:18.927802 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 15:02:18.928709 2658 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:02:20.490123 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:20.490278 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:20.492699 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Sending DeleteTablet for 3 replicas of tablet c1365df26f8f4d33870853b1982ae0a8
I20260927 15:02:20.493301 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet c1365df26f8f4d33870853b1982ae0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55260
I20260927 15:02:20.493466 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet c1365df26f8f4d33870853b1982ae0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48118
I20260927 15:02:20.493515 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet c1365df26f8f4d33870853b1982ae0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39934
I20260927 15:02:20.493850 2716 tablet_replica.cc:330] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: stopping tablet replica
I20260927 15:02:20.494004 2716 raft_consensus.cc:2229] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:20.494340 2716 raft_consensus.cc:2258] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:20.495103 2716 ts_tablet_manager.cc:1905] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:20.496970 2717 tablet_replica.cc:330] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: stopping tablet replica
I20260927 15:02:20.497107 2717 raft_consensus.cc:2229] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:20.497220 2717 raft_consensus.cc:2258] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:20.497450 2715 tablet_replica.cc:330] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: stopping tablet replica
I20260927 15:02:20.497576 2715 raft_consensus.cc:2229] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:20.497680 2715 raft_consensus.cc:2258] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:20.498067 2716 ts_tablet_manager.cc:1918] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:02:20.498080 2717 ts_tablet_manager.cc:1905] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:20.498154 2716 log.cc:1196] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/c1365df26f8f4d33870853b1982ae0a8
I20260927 15:02:20.498283 2715 ts_tablet_manager.cc:1905] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:20.498461 2716 ts_tablet_manager.cc:1939] T c1365df26f8f4d33870853b1982ae0a8 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting consensus metadata
I20260927 15:02:20.499181 2108 catalog_manager.cc:4841] TS 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327): tablet c1365df26f8f4d33870853b1982ae0a8 (table test-table [id=14afc13a65f243cf8687b8e58b03ca8d]) successfully deleted
I20260927 15:02:20.500680 2717 ts_tablet_manager.cc:1918] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:02:20.500789 2717 log.cc:1196] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/c1365df26f8f4d33870853b1982ae0a8
I20260927 15:02:20.500792 2715 ts_tablet_manager.cc:1918] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:02:20.500854 2715 log.cc:1196] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/c1365df26f8f4d33870853b1982ae0a8
I20260927 15:02:20.501066 2717 ts_tablet_manager.cc:1939] T c1365df26f8f4d33870853b1982ae0a8 P d31c2f5e8db545bdac6077d5df4551cc: Deleting consensus metadata
I20260927 15:02:20.501106 2715 ts_tablet_manager.cc:1939] T c1365df26f8f4d33870853b1982ae0a8 P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting consensus metadata
I20260927 15:02:20.501742 2106 catalog_manager.cc:4841] TS d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643): tablet c1365df26f8f4d33870853b1982ae0a8 (table test-table [id=14afc13a65f243cf8687b8e58b03ca8d]) successfully deleted
I20260927 15:02:20.501744 2107 catalog_manager.cc:4841] TS daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623): tablet c1365df26f8f4d33870853b1982ae0a8 (table test-table [id=14afc13a65f243cf8687b8e58b03ca8d]) successfully deleted
Sep 27 15:02:20 dist-test-slave-ml7t krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:02:20 dist-test-slave-ml7t krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:20.531867 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:20.518213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.98.62:37893)
0927 15:02:20.518488 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:20.518494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:20.518548 (+ 54us) server_negotiation.cc:407] Connection header received
0927 15:02:20.518775 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:20.518778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:20.518894 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:20.519064 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:20.519862 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:20.520606 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:20.521445 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:20.521700 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:20.521843 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:20.528476 (+ 6633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:20.528479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:20.528481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:20.528506 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:20.530144 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:20.530624 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:20.530627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:20.530628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:20.530712 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:20.531043 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:20.531046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:20.531048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:20.531255 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:20.531382 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:20.531615 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:02:20.531725 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
I20260927 15:02:20.534590 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:20.535068 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:20.539847 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 62b2239db82f413a9cac6bc63d0e8c39 (DEFAULT_TABLE table=test-table [id=c1733a911ca4478c8535a791d119adc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:20.539882 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 62b2239db82f413a9cac6bc63d0e8c39 (DEFAULT_TABLE table=test-table [id=c1733a911ca4478c8535a791d119adc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:20.540037 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 62b2239db82f413a9cac6bc63d0e8c39 (DEFAULT_TABLE table=test-table [id=c1733a911ca4478c8535a791d119adc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:20.540175 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b2239db82f413a9cac6bc63d0e8c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:20.540184 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b2239db82f413a9cac6bc63d0e8c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:20.540287 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b2239db82f413a9cac6bc63d0e8c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:20.542804 2727 tablet_bootstrap.cc:492] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: Bootstrap starting.
I20260927 15:02:20.543123 2728 tablet_bootstrap.cc:492] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Bootstrap starting.
I20260927 15:02:20.543256 2726 tablet_bootstrap.cc:492] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: Bootstrap starting.
I20260927 15:02:20.543711 2727 tablet_bootstrap.cc:654] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:20.543939 2728 tablet_bootstrap.cc:654] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:20.544137 2726 tablet_bootstrap.cc:654] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:20.544999 2727 tablet_bootstrap.cc:492] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: No bootstrap required, opened a new log
I20260927 15:02:20.545094 2727 ts_tablet_manager.cc:1397] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:20.545326 2726 tablet_bootstrap.cc:492] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: No bootstrap required, opened a new log
I20260927 15:02:20.545408 2726 ts_tablet_manager.cc:1397] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:20.545416 2728 tablet_bootstrap.cc:492] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: No bootstrap required, opened a new log
I20260927 15:02:20.545500 2728 ts_tablet_manager.cc:1397] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:20.545542 2727 raft_consensus.cc:348] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.545680 2727 raft_consensus.cc:374] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:20.545715 2727 raft_consensus.cc:729] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31c2f5e8db545bdac6077d5df4551cc, State: Initialized, Role: FOLLOWER
I20260927 15:02:20.545855 2726 raft_consensus.cc:348] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.545907 2727 consensus_queue.cc:260] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.545984 2726 raft_consensus.cc:374] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:20.546023 2726 raft_consensus.cc:729] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa4ad9e395a4519bacb3ca44e1e1c83, State: Initialized, Role: FOLLOWER
I20260927 15:02:20.545995 2728 raft_consensus.cc:348] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.546120 2728 raft_consensus.cc:374] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:20.546162 2728 raft_consensus.cc:729] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f56f3ecdc42148a9ca8bd7ae5f6e2, State: Initialized, Role: FOLLOWER
I20260927 15:02:20.546170 2726 consensus_queue.cc:260] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.546324 2727 ts_tablet_manager.cc:1428] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:20.546296 2728 consensus_queue.cc:260] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.546561 2726 ts_tablet_manager.cc:1428] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:20.547005 2728 ts_tablet_manager.cc:1428] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:20.865070 2732 raft_consensus.cc:482] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:20.865264 2732 raft_consensus.cc:504] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.865803 2732 leader_election.cc:288] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623), 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327)
I20260927 15:02:20.866300 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b2239db82f413a9cac6bc63d0e8c39" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" is_pre_election: true
I20260927 15:02:20.866297 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b2239db82f413a9cac6bc63d0e8c39" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" is_pre_election: true
I20260927 15:02:20.866482 2545 raft_consensus.cc:2454] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 0.
I20260927 15:02:20.866485 2277 raft_consensus.cc:2454] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 0.
I20260927 15:02:20.866866 2351 leader_election.cc:302] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d31c2f5e8db545bdac6077d5df4551cc, daa4ad9e395a4519bacb3ca44e1e1c83; no voters:
I20260927 15:02:20.867053 2732 raft_consensus.cc:2781] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:20.867110 2732 raft_consensus.cc:482] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:20.867138 2732 raft_consensus.cc:3037] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:20.868137 2732 raft_consensus.cc:504] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.868520 2732 leader_election.cc:288] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 election: Requested vote from peers daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623), 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327)
I20260927 15:02:20.868901 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b2239db82f413a9cac6bc63d0e8c39" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83"
I20260927 15:02:20.868906 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b2239db82f413a9cac6bc63d0e8c39" candidate_uuid: "d31c2f5e8db545bdac6077d5df4551cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2"
I20260927 15:02:20.869043 2277 raft_consensus.cc:3037] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:20.869060 2545 raft_consensus.cc:3037] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:20.870014 2545 raft_consensus.cc:2454] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 1.
I20260927 15:02:20.870172 2277 raft_consensus.cc:2454] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d31c2f5e8db545bdac6077d5df4551cc in term 1.
I20260927 15:02:20.870364 2352 leader_election.cc:302] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, d31c2f5e8db545bdac6077d5df4551cc; no voters:
I20260927 15:02:20.870515 2732 raft_consensus.cc:2781] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:20.870716 2732 raft_consensus.cc:686] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 1 LEADER]: Becoming Leader. State: Replica: d31c2f5e8db545bdac6077d5df4551cc, State: Running, Role: LEADER
I20260927 15:02:20.870873 2732 consensus_queue.cc:237] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.872478 2121 catalog_manager.cc:5495] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc reported cstate change: term changed from 0 to 1, leader changed from <none> to d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d31c2f5e8db545bdac6077d5df4551cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } health_report { overall_health: UNKNOWN } } }
I20260927 15:02:20.918480 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:20.915040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.0.98.2:41643)
0927 15:02:20.915334 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:20.915342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:20.915357 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:02:20.915403 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:20.915405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:20.915452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:20.915538 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:20.915764 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:20.916381 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:20.917345 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:20.917615 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:20.917726 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:20.918015 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:20.918091 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:20.918156 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:20.918243 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 15:02:20.918290 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":144,"threads_started":1}
I20260927 15:02:20.925325 2545 raft_consensus.cc:1261] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Refusing update from remote peer d31c2f5e8db545bdac6077d5df4551cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:20.925341 2277 raft_consensus.cc:1261] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Refusing update from remote peer d31c2f5e8db545bdac6077d5df4551cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:20.925747 2732 consensus_queue.cc:1035] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:20.925927 2733 consensus_queue.cc:1035] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:20.955168 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:20.955300 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:20.957121 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Sending DeleteTablet for 3 replicas of tablet 62b2239db82f413a9cac6bc63d0e8c39
I20260927 15:02:20.957608 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b2239db82f413a9cac6bc63d0e8c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55260
I20260927 15:02:20.957721 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b2239db82f413a9cac6bc63d0e8c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48118
I20260927 15:02:20.957815 2715 tablet_replica.cc:330] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: stopping tablet replica
I20260927 15:02:20.957862 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b2239db82f413a9cac6bc63d0e8c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39934
I20260927 15:02:20.957933 2715 raft_consensus.cc:2229] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:20.958053 2717 tablet_replica.cc:330] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: stopping tablet replica
I20260927 15:02:20.958056 2715 raft_consensus.cc:2258] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:20.958096 2716 tablet_replica.cc:330] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: stopping tablet replica
I20260927 15:02:20.958178 2717 raft_consensus.cc:2229] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:20.958221 2716 raft_consensus.cc:2229] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:20.958359 2716 raft_consensus.cc:2258] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:20.958355 2717 raft_consensus.cc:2258] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:20.958834 2715 ts_tablet_manager.cc:1905] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:20.958858 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:02:20.959028 2716 ts_tablet_manager.cc:1905] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:02:20.959263 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:20.961362 2715 ts_tablet_manager.cc:1918] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:20.961413 2716 ts_tablet_manager.cc:1918] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:20.961454 2715 log.cc:1196] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/62b2239db82f413a9cac6bc63d0e8c39
I20260927 15:02:20.961498 2716 log.cc:1196] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/62b2239db82f413a9cac6bc63d0e8c39
I20260927 15:02:20.961740 2715 ts_tablet_manager.cc:1939] T 62b2239db82f413a9cac6bc63d0e8c39 P daa4ad9e395a4519bacb3ca44e1e1c83: Deleting consensus metadata
I20260927 15:02:20.961802 2716 ts_tablet_manager.cc:1939] T 62b2239db82f413a9cac6bc63d0e8c39 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting consensus metadata
I20260927 15:02:20.962442 2108 catalog_manager.cc:4841] TS 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3:40327): tablet 62b2239db82f413a9cac6bc63d0e8c39 (table test-table [id=c1733a911ca4478c8535a791d119adc2]) successfully deleted
I20260927 15:02:20.962610 2107 catalog_manager.cc:4841] TS daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623): tablet 62b2239db82f413a9cac6bc63d0e8c39 (table test-table [id=c1733a911ca4478c8535a791d119adc2]) successfully deleted
I20260927 15:02:20.962924 2717 ts_tablet_manager.cc:1905] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:20.964193 2538 tablet_service.cc:1467] Processing CreateTablet for tablet c9be3c5b288744d694cb0c28f1c026c1 (DEFAULT_TABLE table=test-table [id=22890c398879444597c36db10f9684ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:20.964468 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9be3c5b288744d694cb0c28f1c026c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:20.965379 2717 ts_tablet_manager.cc:1918] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:20.965466 2717 log.cc:1196] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/62b2239db82f413a9cac6bc63d0e8c39
I20260927 15:02:20.965747 2717 ts_tablet_manager.cc:1939] T 62b2239db82f413a9cac6bc63d0e8c39 P d31c2f5e8db545bdac6077d5df4551cc: Deleting consensus metadata
I20260927 15:02:20.966077 2398 tablet_service.cc:1467] Processing CreateTablet for tablet c9be3c5b288744d694cb0c28f1c026c1 (DEFAULT_TABLE table=test-table [id=22890c398879444597c36db10f9684ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:20.966331 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9be3c5b288744d694cb0c28f1c026c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:20.966414 2106 catalog_manager.cc:4841] TS d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643): tablet 62b2239db82f413a9cac6bc63d0e8c39 (table test-table [id=c1733a911ca4478c8535a791d119adc2]) successfully deleted
I20260927 15:02:20.966723 2728 tablet_bootstrap.cc:492] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Bootstrap starting.
I20260927 15:02:20.967546 2728 tablet_bootstrap.cc:654] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:20.968600 2727 tablet_bootstrap.cc:492] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc: Bootstrap starting.
I20260927 15:02:20.968657 2728 tablet_bootstrap.cc:492] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: No bootstrap required, opened a new log
I20260927 15:02:20.968730 2728 ts_tablet_manager.cc:1397] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:20.969128 2728 raft_consensus.cc:348] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.969172 2258 tablet_service.cc:1467] Processing CreateTablet for tablet c9be3c5b288744d694cb0c28f1c026c1 (DEFAULT_TABLE table=test-table [id=22890c398879444597c36db10f9684ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:20.969250 2728 raft_consensus.cc:374] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:20.969282 2728 raft_consensus.cc:729] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f56f3ecdc42148a9ca8bd7ae5f6e2, State: Initialized, Role: FOLLOWER
I20260927 15:02:20.969417 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9be3c5b288744d694cb0c28f1c026c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:20.969416 2728 consensus_queue.cc:260] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.969549 2727 tablet_bootstrap.cc:654] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:20.969713 2728 ts_tablet_manager.cc:1428] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:20.970728 2727 tablet_bootstrap.cc:492] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc: No bootstrap required, opened a new log
I20260927 15:02:20.970829 2727 ts_tablet_manager.cc:1397] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 15:02:20.971211 2727 raft_consensus.cc:348] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.971328 2727 raft_consensus.cc:374] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:20.971380 2727 raft_consensus.cc:729] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31c2f5e8db545bdac6077d5df4551cc, State: Initialized, Role: FOLLOWER
I20260927 15:02:20.971522 2727 consensus_queue.cc:260] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.971807 2727 ts_tablet_manager.cc:1428] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:02:20.972720 2726 tablet_bootstrap.cc:492] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83: Bootstrap starting.
I20260927 15:02:20.973604 2726 tablet_bootstrap.cc:654] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:20.975008 2726 tablet_bootstrap.cc:492] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83: No bootstrap required, opened a new log
I20260927 15:02:20.975111 2726 ts_tablet_manager.cc:1397] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:20.975453 2726 raft_consensus.cc:348] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.975561 2726 raft_consensus.cc:374] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:20.975605 2726 raft_consensus.cc:729] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa4ad9e395a4519bacb3ca44e1e1c83, State: Initialized, Role: FOLLOWER
I20260927 15:02:20.975731 2726 consensus_queue.cc:260] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.976017 2726 ts_tablet_manager.cc:1428] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:20.989195 2710 raft_consensus.cc:482] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:20.989326 2710 raft_consensus.cc:504] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.989847 2710 leader_election.cc:288] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623), d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643)
I20260927 15:02:20.990314 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9be3c5b288744d694cb0c28f1c026c1" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" is_pre_election: true
I20260927 15:02:20.990299 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9be3c5b288744d694cb0c28f1c026c1" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31c2f5e8db545bdac6077d5df4551cc" is_pre_election: true
I20260927 15:02:20.990484 2277 raft_consensus.cc:2454] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 0.
I20260927 15:02:20.990492 2418 raft_consensus.cc:2454] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 0.
I20260927 15:02:20.990808 2491 leader_election.cc:302] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, daa4ad9e395a4519bacb3ca44e1e1c83; no voters:
I20260927 15:02:20.991014 2710 raft_consensus.cc:2781] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:20.991070 2710 raft_consensus.cc:482] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:20.991132 2710 raft_consensus.cc:3037] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:20.992110 2710 raft_consensus.cc:504] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.992470 2710 leader_election.cc:288] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 election: Requested vote from peers daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623), d31c2f5e8db545bdac6077d5df4551cc (127.0.98.2:41643)
I20260927 15:02:20.992755 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9be3c5b288744d694cb0c28f1c026c1" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83"
I20260927 15:02:20.992844 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9be3c5b288744d694cb0c28f1c026c1" candidate_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31c2f5e8db545bdac6077d5df4551cc"
I20260927 15:02:20.992883 2277 raft_consensus.cc:3037] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:20.992960 2418 raft_consensus.cc:3037] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:20.994077 2418 raft_consensus.cc:2454] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 1.
I20260927 15:02:20.994077 2277 raft_consensus.cc:2454] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f56f3ecdc42148a9ca8bd7ae5f6e2 in term 1.
I20260927 15:02:20.994364 2491 leader_election.cc:302] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f56f3ecdc42148a9ca8bd7ae5f6e2, daa4ad9e395a4519bacb3ca44e1e1c83; no voters:
I20260927 15:02:20.994567 2710 raft_consensus.cc:2781] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:20.994773 2710 raft_consensus.cc:686] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 LEADER]: Becoming Leader. State: Replica: 449f56f3ecdc42148a9ca8bd7ae5f6e2, State: Running, Role: LEADER
I20260927 15:02:20.994957 2710 consensus_queue.cc:237] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } }
I20260927 15:02:20.996634 2120 catalog_manager.cc:5495] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 449f56f3ecdc42148a9ca8bd7ae5f6e2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449f56f3ecdc42148a9ca8bd7ae5f6e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40327 } health_report { overall_health: HEALTHY } } }
I20260927 15:02:21.018816 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:21.015567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.0.98.3:40327)
0927 15:02:21.015772 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:21.015795 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:21.015810 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:02:21.015853 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:21.015856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:21.015907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:21.015999 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:21.016234 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:21.016804 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:21.017636 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:21.017979 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:21.018089 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:21.018335 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:21.018422 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:21.018503 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:21.018604 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 15:02:21.018655 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260927 15:02:21.024560 2277 raft_consensus.cc:1261] T c9be3c5b288744d694cb0c28f1c026c1 P daa4ad9e395a4519bacb3ca44e1e1c83 [term 1 FOLLOWER]: Refusing update from remote peer 449f56f3ecdc42148a9ca8bd7ae5f6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:21.024902 2417 raft_consensus.cc:1261] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Refusing update from remote peer 449f56f3ecdc42148a9ca8bd7ae5f6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:21.025285 2744 consensus_queue.cc:1035] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31c2f5e8db545bdac6077d5df4551cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:21.025424 2710 consensus_queue.cc:1035] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa4ad9e395a4519bacb3ca44e1e1c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:02:21.043476 2697 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:02:22.594204 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:22.594410 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:22.596879 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59e8b51a54c041b2a16428f00d4081ca: Sending DeleteTablet for 3 replicas of tablet c9be3c5b288744d694cb0c28f1c026c1
I20260927 15:02:22.597487 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet c9be3c5b288744d694cb0c28f1c026c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39934
I20260927 15:02:22.597522 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet c9be3c5b288744d694cb0c28f1c026c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48118
I20260927 15:02:22.597834 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet c9be3c5b288744d694cb0c28f1c026c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55260
I20260927 15:02:22.597883 2772 tablet_replica.cc:330] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc: stopping tablet replica
I20260927 15:02:22.597960 2773 tablet_replica.cc:330] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: stopping tablet replica
I20260927 15:02:22.598024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 2193
I20260927 15:02:22.598840 2773 raft_consensus.cc:2229] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:22.599076 2773 raft_consensus.cc:2258] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:22.599511 2772 raft_consensus.cc:2229] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:22.599661 2772 raft_consensus.cc:2258] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:22.600031 2773 ts_tablet_manager.cc:1905] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:22.600417 2772 ts_tablet_manager.cc:1905] T c9be3c5b288744d694cb0c28f1c026c1 P d31c2f5e8db545bdac6077d5df4551cc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:02:22.606508 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36623 (error 108)
I20260927 15:02:22.606635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 2333
I20260927 15:02:22.608767 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:22.608231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.0.1:55640)
0927 15:02:22.608556 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:22.608637 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":88,"threads_started":1}
W20260927 15:02:22.608939 2107 catalog_manager.cc:4584] TS daa4ad9e395a4519bacb3ca44e1e1c83 (127.0.98.1:36623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9be3c5b288744d694cb0c28f1c026c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36623: connect: Connection refused (error 111)
I20260927 15:02:22.658530 2773 ts_tablet_manager.cc:1918] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:02:22.658658 2773 log.cc:1196] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/c9be3c5b288744d694cb0c28f1c026c1
I20260927 15:02:22.659003 2773 ts_tablet_manager.cc:1939] T c9be3c5b288744d694cb0c28f1c026c1 P 449f56f3ecdc42148a9ca8bd7ae5f6e2: Deleting consensus metadata
I20260927 15:02:22.661733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 2473
I20260927 15:02:22.662664 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:22.661914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.0.1:55656)
0927 15:02:22.662463 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:22.662547 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
I20260927 15:02:22.663561 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:22.663245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36623 (local address 127.0.0.1:55666)
0927 15:02:22.663415 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:22.663475 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 15:02:22.663935 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:22.663695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41643 (local address 127.0.0.1:53156)
0927 15:02:22.663809 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:22.663889 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:02:22.696215 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:22.695820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41643 (local address 127.0.0.1:53168)
0927 15:02:22.696059 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:22.696133 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 15:02:22.697106 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:22.696830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41643 (local address 127.0.0.1:53170)
0927 15:02:22.696999 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:22.697051 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 15:02:22.711548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 2089
2026-09-27T15:02:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6599 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:23 dist-test-slave-ml7t krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:23 dist-test-slave-ml7t krb5kdc[2779](info): set up 2 sockets
Sep 27 15:02:23 dist-test-slave-ml7t krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260927 15:02:24.737623 392 mini_kdc.cc:121] Time spent starting KDC: real 2.003s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:24 dist-test-slave-ml7t krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:24.860723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:24.971628 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:24.971966 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:24.972043 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:24.976148 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:24.976260 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:24.976300 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:24.976361 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:24.976399 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:24.981806 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:24.982944 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:24.983968 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:24.993001 2792 server_base.cc:1034] running on GCE node
W20260927 15:02:24.993054 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:24.993042 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:24.993043 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:24.993901 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:24.995012 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:24.996203 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790521344996182 us; error 34 us; skew 500 ppm
Sep 27 15:02:24 dist-test-slave-ml7t krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:24.998983 2792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:24.999871 2792 webserver.cc:460] Webserver started at http://127.0.98.62:35465/ using document root <none> and password file <none>
I20260927 15:02:25.000393 2792 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:25.000460 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:25.000661 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:25.002681 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "c209df87b8754b8a8720db9bf16ef5bf"
format_stamp: "Formatted at 2026-09-27 15:02:24 on dist-test-slave-ml7t"
server_key: "c200038d9d707c6638764e68a060e13b"
server_key_iv: "fcf3f73a35ae5c39aa078bf87a7fd891"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.003196 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "c209df87b8754b8a8720db9bf16ef5bf"
format_stamp: "Formatted at 2026-09-27 15:02:24 on dist-test-slave-ml7t"
server_key: "c200038d9d707c6638764e68a060e13b"
server_key_iv: "fcf3f73a35ae5c39aa078bf87a7fd891"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.006904 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:25.009323 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.010655 2792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:25.010766 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "c209df87b8754b8a8720db9bf16ef5bf"
format_stamp: "Formatted at 2026-09-27 15:02:24 on dist-test-slave-ml7t"
server_key: "c200038d9d707c6638764e68a060e13b"
server_key_iv: "fcf3f73a35ae5c39aa078bf87a7fd891"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.010875 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:25.019887 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:25.022902 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:25.023051 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:25.030007 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43309
I20260927 15:02:25.030045 2859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43309 every 8 connection(s)
I20260927 15:02:25.030964 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:25.033578 2860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:25.037223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 2792
I20260927 15:02:25.037338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:25.037624 392 external_mini_cluster.cc:1428] Setting key e82a29a7b75a564c125c64428a4acb11
I20260927 15:02:25.039544 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf: Bootstrap starting.
I20260927 15:02:25.041615 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:25.042361 2860 log.cc:824] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:25.044564 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf: No bootstrap required, opened a new log
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:25.049620 2860 raft_consensus.cc:348] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c209df87b8754b8a8720db9bf16ef5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43309 } }
I20260927 15:02:25.049968 2860 raft_consensus.cc:374] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:25.050037 2860 raft_consensus.cc:729] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c209df87b8754b8a8720db9bf16ef5bf, State: Initialized, Role: FOLLOWER
I20260927 15:02:25.050546 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c209df87b8754b8a8720db9bf16ef5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43309 } }
I20260927 15:02:25.050724 2860 raft_consensus.cc:388] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:25.050804 2860 raft_consensus.cc:482] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:25.050911 2860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:25.052081 2860 raft_consensus.cc:504] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c209df87b8754b8a8720db9bf16ef5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43309 } }
I20260927 15:02:25.052433 2860 leader_election.cc:302] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c209df87b8754b8a8720db9bf16ef5bf; no voters:
I20260927 15:02:25.052800 2860 leader_election.cc:288] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:25.052906 2865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:25.053015 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:25.039109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.0.98.62:43309)
0927 15:02:25.039521 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:25.039536 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:25.039572 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:02:25.040234 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:25.040250 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:25.040578 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:25.040917 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:25.041694 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.042898 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.044275 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.044793 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.044931 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:25.047708 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:25.047717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:25.047739 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:25.047775 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:25.049739 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.050289 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.050294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.050300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.050430 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.050688 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.050691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.050693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.051274 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:25.051453 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:25.051706 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:02:25.052018 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
I20260927 15:02:25.053164 2865 raft_consensus.cc:686] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [term 1 LEADER]: Becoming Leader. State: Replica: c209df87b8754b8a8720db9bf16ef5bf, State: Running, Role: LEADER
I20260927 15:02:25.053485 2865 consensus_queue.cc:237] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c209df87b8754b8a8720db9bf16ef5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43309 } }
I20260927 15:02:25.054075 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:25.054755 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c209df87b8754b8a8720db9bf16ef5bf. Latest consensus state: current_term: 1 leader_uuid: "c209df87b8754b8a8720db9bf16ef5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c209df87b8754b8a8720db9bf16ef5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43309 } } }
I20260927 15:02:25.054881 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:25.055500 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c209df87b8754b8a8720db9bf16ef5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c209df87b8754b8a8720db9bf16ef5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43309 } } }
I20260927 15:02:25.055583 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf [sys.catalog]: This master's current role is: LEADER
W20260927 15:02:25.058604 2880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:02:25.058665 2880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:02:25.058740 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:25.059605 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:25.065321 2873 catalog_manager.cc:1347] Generated new cluster ID: ab1e93cf2a6348a180deab8d0c66e702
I20260927 15:02:25.065402 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:25.089258 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:25.090591 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:25.102442 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c209df87b8754b8a8720db9bf16ef5bf: Generated new TSK 0
I20260927 15:02:25.103178 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:25.137267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43309
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:25.258780 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:25.259068 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:25.259114 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:25.263365 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:25.263432 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:25.263494 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:25.268848 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:25.269891 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:25.270864 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:25.281255 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:25.281263 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:25.281267 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:25.281636 2886 server_base.cc:1034] running on GCE node
I20260927 15:02:25.282049 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:25.282706 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:25.283903 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790521345283871 us; error 31 us; skew 500 ppm
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:25.286635 2886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:25.287463 2886 webserver.cc:460] Webserver started at http://127.0.98.1:37249/ using document root <none> and password file <none>
I20260927 15:02:25.288015 2886 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:25.288103 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:25.288326 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:25.290359 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c0885e42a354b7bbe7bb3cdfc72d4ce"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "71673844b7466c9549581d6067f7b36c"
server_key_iv: "6dadd269092b5193588960478d78705e"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.290917 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c0885e42a354b7bbe7bb3cdfc72d4ce"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "71673844b7466c9549581d6067f7b36c"
server_key_iv: "6dadd269092b5193588960478d78705e"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.294525 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:25.296806 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.298081 2886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:25.298204 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "4c0885e42a354b7bbe7bb3cdfc72d4ce"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "71673844b7466c9549581d6067f7b36c"
server_key_iv: "6dadd269092b5193588960478d78705e"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.298306 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:25.319164 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:25.322446 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:25.322647 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:25.323248 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:25.324110 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:25.324174 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.324240 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:25.324294 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.333820 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39299
I20260927 15:02:25.333881 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39299 every 8 connection(s)
I20260927 15:02:25.334800 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:25.344514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 2886
I20260927 15:02:25.344650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:25.344919 392 external_mini_cluster.cc:1428] Setting key 5b4d126e9d6c46bf6372374a4ddd9946
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:25.350019 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:25.336672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51379 (local address 127.0.98.62:43309)
0927 15:02:25.336798 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:25.336802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:25.337649 (+ 847us) server_negotiation.cc:407] Connection header received
0927 15:02:25.339100 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:25.339107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:25.339151 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:25.339228 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:25.340356 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.341288 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.342667 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.342935 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.343027 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:25.346029 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:25.346040 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:25.346042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:25.346068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:25.347589 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.348235 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.348238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.348240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.348328 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.348720 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.348723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.348725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.348928 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:25.349041 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:25.349687 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 15:02:25.349887 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:02:25.351516 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:25.337067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43309 (local address 127.0.98.1:51379)
0927 15:02:25.337491 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:25.337533 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:25.338889 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:25.339357 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:25.339363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:25.339709 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:25.340201 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:25.340215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.341443 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.341449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:25.342531 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:25.342541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.343080 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.343083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:25.343231 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:25.343872 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:25.343895 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:25.345755 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:25.347734 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:25.347741 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:25.347762 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:25.348076 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:25.348427 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:25.348431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:25.348433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:25.348608 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:25.349041 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:25.349050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:25.349358 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 15:02:25.350223 (+ 865us) client_negotiation.cc:240] Negotiation successful
0927 15:02:25.350630 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260927 15:02:25.353334 3015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43309
I20260927 15:02:25.353642 3015 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:02:25.354310 3015 heartbeater.cc:511] Master 127.0.98.62:43309 requested a full tablet report, sending...
I20260927 15:02:25.356099 2823 ts_manager.cc:194] Registered new tserver with Master: 4c0885e42a354b7bbe7bb3cdfc72d4ce (127.0.98.1:39299)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:25.357501 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:25.373502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43309
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:25.483089 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:25.483345 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:25.483387 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:25.487494 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:25.487555 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:25.487650 3020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:25.493031 3020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:25.494108 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:25.495111 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:25.504736 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:25.504909 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:25.504937 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:25.505039 3020 server_base.cc:1034] running on GCE node
I20260927 15:02:25.505515 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:25.506161 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:25.507368 3020 hybrid_clock.cc:648] HybridClock initialized: now 1790521345507351 us; error 27 us; skew 500 ppm
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:25.510109 3020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:25.510917 3020 webserver.cc:460] Webserver started at http://127.0.98.2:35981/ using document root <none> and password file <none>
I20260927 15:02:25.511458 3020 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:25.511545 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:25.511770 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:25.513711 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "45b86f86f2f049199fa18ab98eb9a2ae"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "0f67b1940a7647269881e74081f52345"
server_key_iv: "d627765d647e701000188b7169f0c2e7"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.514349 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "45b86f86f2f049199fa18ab98eb9a2ae"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "0f67b1940a7647269881e74081f52345"
server_key_iv: "d627765d647e701000188b7169f0c2e7"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.518013 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:25.520299 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.521585 3020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:25.521682 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "45b86f86f2f049199fa18ab98eb9a2ae"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "0f67b1940a7647269881e74081f52345"
server_key_iv: "d627765d647e701000188b7169f0c2e7"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.521871 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:25.556953 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:25.560294 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:25.560508 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:25.561116 3020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:25.562008 3020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:25.562078 3020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.562165 3020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:25.562215 3020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.570735 3020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46489
I20260927 15:02:25.570801 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46489 every 8 connection(s)
I20260927 15:02:25.571666 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:25.580008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3020
I20260927 15:02:25.580093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:25.580344 392 external_mini_cluster.cc:1428] Setting key 254d9bbe205c6d0cb2abcd6aabdf096f
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:25.586211 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:25.573459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44347 (local address 127.0.98.62:43309)
0927 15:02:25.573619 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:25.573624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:25.574515 (+ 891us) server_negotiation.cc:407] Connection header received
0927 15:02:25.575822 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:25.575825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:25.575868 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:25.575946 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:25.577208 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.577730 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.578864 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.579076 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.579155 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:25.582116 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:25.582120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:25.582126 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:25.582158 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:25.583804 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.584407 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.584409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.584411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.584485 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.584921 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.584924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.584925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.585130 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:25.585238 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:25.585914 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 15:02:25.586023 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:02:25.587298 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:25.573928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43309 (local address 127.0.98.2:44347)
0927 15:02:25.574361 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:25.574404 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:25.575627 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:25.576142 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:25.576149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:25.576506 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:25.577068 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:25.577084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.577899 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.577902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:25.578757 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:25.578766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.579200 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.579202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:25.579339 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:25.580004 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:25.580021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:25.581953 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:25.583914 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:25.583923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:25.583938 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:25.584298 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:25.584581 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:25.584585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:25.584587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:25.584767 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:25.585244 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:25.585253 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:25.585585 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 15:02:25.585827 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 15:02:25.586200 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20260927 15:02:25.588570 3149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43309
I20260927 15:02:25.588856 3149 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:25.589574 3149 heartbeater.cc:511] Master 127.0.98.62:43309 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:25.590845 2823 ts_manager.cc:194] Registered new tserver with Master: 45b86f86f2f049199fa18ab98eb9a2ae (127.0.98.2:46489)
I20260927 15:02:25.591470 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:25.609062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43309
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:25.717810 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:25.718080 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:25.718153 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:25.722118 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:25.722189 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:25.722285 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:25.728072 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:25.729033 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:25.730288 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:25.739976 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:25.739977 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:25.740061 3154 server_base.cc:1034] running on GCE node
W20260927 15:02:25.739979 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:25.740666 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:25.741400 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:25.742595 3154 hybrid_clock.cc:648] HybridClock initialized: now 1790521345742578 us; error 27 us; skew 500 ppm
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:25.745503 3154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:25.746395 3154 webserver.cc:460] Webserver started at http://127.0.98.3:44209/ using document root <none> and password file <none>
I20260927 15:02:25.746945 3154 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:25.747035 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:25.747263 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:25.749243 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "742ae940ecc54f55ae41a0fc668a24d1"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "a2eabfc6a038d80fd403a64fbebe8173"
server_key_iv: "00c3288de01bd6a8260a565d1b49da11"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.749859 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "742ae940ecc54f55ae41a0fc668a24d1"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "a2eabfc6a038d80fd403a64fbebe8173"
server_key_iv: "00c3288de01bd6a8260a565d1b49da11"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.753489 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:25.756062 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.757217 3154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:25.757355 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "742ae940ecc54f55ae41a0fc668a24d1"
format_stamp: "Formatted at 2026-09-27 15:02:25 on dist-test-slave-ml7t"
server_key: "a2eabfc6a038d80fd403a64fbebe8173"
server_key_iv: "00c3288de01bd6a8260a565d1b49da11"
server_key_version: "encryptionkey@0"
I20260927 15:02:25.757459 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:25.784976 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:25.788192 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:25.788403 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:25.788987 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:25.789875 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:25.789947 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.790023 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:25.790060 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:25.799362 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33871
I20260927 15:02:25.799381 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33871 every 8 connection(s)
I20260927 15:02:25.800284 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:25.805168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3154
I20260927 15:02:25.805294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:25.805581 392 external_mini_cluster.cc:1428] Setting key 88c095ec8a12f225fe298c659494ab59
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:25.814007 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:25.802143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33817 (local address 127.0.98.62:43309)
0927 15:02:25.802313 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:25.802318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:25.803143 (+ 825us) server_negotiation.cc:407] Connection header received
0927 15:02:25.804487 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:25.804490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:25.804535 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:25.804613 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:25.806166 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.806806 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.807937 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.808174 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.808285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:25.810660 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:25.810664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:25.810666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:25.810692 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:25.811955 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.812449 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.812453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.812455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.812542 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:25.812872 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:25.812876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:25.812877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:25.813053 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:25.813157 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:25.813686 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 15:02:25.813840 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:02:25.814857 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:25.802550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43309 (local address 127.0.98.3:33817)
0927 15:02:25.802992 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:25.803033 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:25.804248 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:25.804779 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:25.804786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:25.805312 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:25.806019 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:25.806037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.806939 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.806943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:25.807810 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:25.807819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:25.808318 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:25.808322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:25.808491 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:25.809094 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:25.809110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:25.810495 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:25.812071 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:25.812077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:25.812091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:25.812344 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:25.812632 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:25.812636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:25.812638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:25.812770 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:25.813155 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:25.813161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:25.813443 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 15:02:25.813636 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 15:02:25.813963 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260927 15:02:25.815958 3283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43309
I20260927 15:02:25.816190 3283 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:25.816653 3283 heartbeater.cc:511] Master 127.0.98.62:43309 requested a full tablet report, sending...
I20260927 15:02:25.817588 2823 ts_manager.cc:194] Registered new tserver with Master: 742ae940ecc54f55ae41a0fc668a24d1 (127.0.98.3:33871)
I20260927 15:02:25.818329 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33817
I20260927 15:02:25.819130 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:02:25.831447 2863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:25.830111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.98.62:43309)
0927 15:02:25.830231 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:25.830235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:25.830417 (+ 182us) server_negotiation.cc:407] Connection header received
0927 15:02:25.830522 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:25.830524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:25.830562 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:25.830639 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:25.831302 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47832: BlockingRecv error: recv got EOF from 127.0.0.1:47832 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:02:25.832522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 2886
I20260927 15:02:25.837970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3020
I20260927 15:02:25.843065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3154
I20260927 15:02:25.848011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 2792
2026-09-27T15:02:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3135 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[3295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[3295](info): set up 2 sockets
Sep 27 15:02:25 dist-test-slave-ml7t krb5kdc[3295](info): commencing operation
krb5kdc: starting...
W20260927 15:02:26.885703 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:26 dist-test-slave-ml7t krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:27.005252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:27.113307 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:27.113583 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:27.113623 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:27.117622 3308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:27.117695 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:27.117715 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:27.117731 3308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:27.117748 3308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:27.123026 3308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:27.124014 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:27.124958 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:27.133344 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:27.133350 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:27.133409 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:27.133497 3308 server_base.cc:1034] running on GCE node
I20260927 15:02:27.134176 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:27.135223 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:27.136394 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790521347136380 us; error 27 us; skew 500 ppm
Sep 27 15:02:27 dist-test-slave-ml7t krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:27.139369 3308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:27.140249 3308 webserver.cc:460] Webserver started at http://127.0.98.62:43977/ using document root <none> and password file <none>
I20260927 15:02:27.140792 3308 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:27.140853 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:27.141069 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:27.143133 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "a6c8a91eaef449ae95421083a25a1c53"
format_stamp: "Formatted at 2026-09-27 15:02:27 on dist-test-slave-ml7t"
server_key: "7619058c51e7ad0812df7f270deaebcd"
server_key_iv: "c6cef94d17e2e25b6181f60da650527d"
server_key_version: "encryptionkey@0"
I20260927 15:02:27.143688 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6c8a91eaef449ae95421083a25a1c53"
format_stamp: "Formatted at 2026-09-27 15:02:27 on dist-test-slave-ml7t"
server_key: "7619058c51e7ad0812df7f270deaebcd"
server_key_iv: "c6cef94d17e2e25b6181f60da650527d"
server_key_version: "encryptionkey@0"
I20260927 15:02:27.147274 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:02:27.149633 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:27.150919 3308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:27.151017 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "a6c8a91eaef449ae95421083a25a1c53"
format_stamp: "Formatted at 2026-09-27 15:02:27 on dist-test-slave-ml7t"
server_key: "7619058c51e7ad0812df7f270deaebcd"
server_key_iv: "c6cef94d17e2e25b6181f60da650527d"
server_key_version: "encryptionkey@0"
I20260927 15:02:27.151245 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:27.167359 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:27.170691 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:27.170908 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:27.177888 3308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37765
I20260927 15:02:27.178279 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37765 every 8 connection(s)
I20260927 15:02:27.178925 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:27.181335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3308
I20260927 15:02:27.181430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:27.181699 392 external_mini_cluster.cc:1428] Setting key 5c332fa67bcd872238f5550d27c0c1e7
I20260927 15:02:27.181821 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:27.189402 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53: Bootstrap starting.
Sep 27 15:02:27 dist-test-slave-ml7t krb5kdc[3295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:27.192411 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:27.193346 3376 log.cc:824] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:27.195785 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53: No bootstrap required, opened a new log
I20260927 15:02:27.197713 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:27.183069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.0.98.62:37765)
0927 15:02:27.183483 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:27.183500 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:27.183536 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:02:27.184490 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:27.184509 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:27.184851 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:27.185150 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:27.186326 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:27.187500 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:27.188784 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:27.189273 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:27.189406 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:27.192473 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:27.192479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:27.192492 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:27.192521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:27.194786 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:27.195310 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:27.195315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:27.195320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:27.195435 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:27.195736 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:27.195738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:27.195739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:27.196212 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:27.196367 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:27.196668 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 15:02:27.197003 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":147,"threads_started":1}
I20260927 15:02:27.199950 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c8a91eaef449ae95421083a25a1c53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260927 15:02:27.200191 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:27.200285 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c8a91eaef449ae95421083a25a1c53, State: Initialized, Role: FOLLOWER
I20260927 15:02:27.200691 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:02:27.200798 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c8a91eaef449ae95421083a25a1c53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260927 15:02:27.200953 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:27.201010 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:27.201099 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:27.202589 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c8a91eaef449ae95421083a25a1c53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260927 15:02:27.202981 3376 leader_election.cc:302] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c8a91eaef449ae95421083a25a1c53; no voters:
I20260927 15:02:27.203341 3376 leader_election.cc:288] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:27.203423 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:27.203600 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [term 1 LEADER]: Becoming Leader. State: Replica: a6c8a91eaef449ae95421083a25a1c53, State: Running, Role: LEADER
I20260927 15:02:27.203948 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c8a91eaef449ae95421083a25a1c53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260927 15:02:27.204640 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:27.205158 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c8a91eaef449ae95421083a25a1c53. Latest consensus state: current_term: 1 leader_uuid: "a6c8a91eaef449ae95421083a25a1c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c8a91eaef449ae95421083a25a1c53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } } }
I20260927 15:02:27.205341 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:27.205600 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c8a91eaef449ae95421083a25a1c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c8a91eaef449ae95421083a25a1c53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } } }
I20260927 15:02:27.205816 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:27.205843 3388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:27.209204 3388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:27.216032 3388 catalog_manager.cc:1347] Generated new cluster ID: 84f3ff7f6d6c4ed98b99ac49e351f6db
I20260927 15:02:27.216176 3388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:27.223070 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:27.217416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.0.98.62:37765)
0927 15:02:27.217539 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:27.217544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:27.217822 (+ 278us) server_negotiation.cc:407] Connection header received
0927 15:02:27.217965 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:27.217968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:27.218020 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:27.218123 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:27.218667 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:27.219517 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:27.220661 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:27.221024 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:27.221131 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:27.222046 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:27.222050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:27.222053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:27.222589 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:27.222622 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:27.222643 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:27.222736 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:27.222874 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:02:27.222928 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:02:27.224318 3388 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:27.225142 3388 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:27.230949 3388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6c8a91eaef449ae95421083a25a1c53: Generated new TSK 0
I20260927 15:02:27.231801 3388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:02:27.244246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3308
2026-09-27T15:02:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1394 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:27 dist-test-slave-ml7t krb5kdc[3409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:27 dist-test-slave-ml7t krb5kdc[3409](info): set up 2 sockets
Sep 27 15:02:27 dist-test-slave-ml7t krb5kdc[3409](info): commencing operation
krb5kdc: starting...
W20260927 15:02:28.293684 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:28 dist-test-slave-ml7t krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:28.413004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:28.528703 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:28.528988 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:28.529060 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:28.533238 3422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:28.533304 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:28.533324 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:28.533341 3422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:28.533382 3422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:28.538852 3422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:28.539929 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:28.540967 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:28.549837 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:28.549836 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:28.549921 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:28.550061 3422 server_base.cc:1034] running on GCE node
I20260927 15:02:28.550623 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:28.551720 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:28.552932 3422 hybrid_clock.cc:648] HybridClock initialized: now 1790521348552919 us; error 26 us; skew 500 ppm
Sep 27 15:02:28 dist-test-slave-ml7t krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:28.555909 3422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:28.556896 3422 webserver.cc:460] Webserver started at http://127.0.98.62:37155/ using document root <none> and password file <none>
I20260927 15:02:28.557458 3422 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:28.557548 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:28.557825 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:28.559918 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e"
format_stamp: "Formatted at 2026-09-27 15:02:28 on dist-test-slave-ml7t"
server_key: "e3267f0fd138380ee6a2f8c93f37dda0"
server_key_iv: "e70bd83172cf7c6b288e7cc9e48977c8"
server_key_version: "encryptionkey@0"
I20260927 15:02:28.560500 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e"
format_stamp: "Formatted at 2026-09-27 15:02:28 on dist-test-slave-ml7t"
server_key: "e3267f0fd138380ee6a2f8c93f37dda0"
server_key_iv: "e70bd83172cf7c6b288e7cc9e48977c8"
server_key_version: "encryptionkey@0"
I20260927 15:02:28.564448 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 15:02:28.567166 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:28.568631 3422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:02:28.568810 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e"
format_stamp: "Formatted at 2026-09-27 15:02:28 on dist-test-slave-ml7t"
server_key: "e3267f0fd138380ee6a2f8c93f37dda0"
server_key_iv: "e70bd83172cf7c6b288e7cc9e48977c8"
server_key_version: "encryptionkey@0"
I20260927 15:02:28.568933 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:28.586316 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:28.589143 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:28.589347 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:28.596097 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46345
I20260927 15:02:28.596395 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46345 every 8 connection(s)
I20260927 15:02:28.597086 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:28.598819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3422
I20260927 15:02:28.598945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:28.599233 392 external_mini_cluster.cc:1428] Setting key c90c5525fb121224cc88d2e3151df78a
I20260927 15:02:28.600155 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:28.607101 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e: Bootstrap starting.
Sep 27 15:02:28 dist-test-slave-ml7t krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:28.609920 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:28.610875 3490 log.cc:824] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:28.613590 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e: No bootstrap required, opened a new log
I20260927 15:02:28.616588 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:28.600601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.0.98.62:46345)
0927 15:02:28.601022 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:28.601041 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:28.601083 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:02:28.602011 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:28.602036 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:28.602459 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:28.602832 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:28.603494 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:28.604962 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:28.606051 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:28.606738 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:28.606911 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:28.609743 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:28.609752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:28.609791 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:28.609834 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:28.612167 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:28.612784 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:28.612790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:28.612796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:28.612954 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:28.613340 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:28.613345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:28.613347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:28.613985 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:28.614873 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:28.615047 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:02:28.615415 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":124,"threads_started":1}
I20260927 15:02:28.618053 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46345 } }
I20260927 15:02:28.618324 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:28.618434 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfb7b67ddc74c7b807bb27cc47aa03e, State: Initialized, Role: FOLLOWER
I20260927 15:02:28.618911 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46345 } }
I20260927 15:02:28.619092 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:28.619159 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:28.619252 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:28.620561 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46345 } }
I20260927 15:02:28.620939 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cfb7b67ddc74c7b807bb27cc47aa03e; no voters:
I20260927 15:02:28.621260 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:28.621446 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:28.621728 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [term 1 LEADER]: Becoming Leader. State: Replica: 1cfb7b67ddc74c7b807bb27cc47aa03e, State: Running, Role: LEADER
I20260927 15:02:28.622112 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46345 } }
I20260927 15:02:28.622555 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:28.623736 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46345 } } }
I20260927 15:02:28.623875 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:28.624291 3504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:28.624238 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cfb7b67ddc74c7b807bb27cc47aa03e. Latest consensus state: current_term: 1 leader_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cfb7b67ddc74c7b807bb27cc47aa03e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46345 } } }
I20260927 15:02:28.624436 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:28.627565 3504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:28.633852 3504 catalog_manager.cc:1347] Generated new cluster ID: de3c69653f044e3d9039bc49110a80c3
I20260927 15:02:28.633957 3504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:28.662393 3504 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:28.663368 3504 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:28.668478 3504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cfb7b67ddc74c7b807bb27cc47aa03e: Generated new TSK 0
I20260927 15:02:28.669327 3504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:28.706609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46345
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:28.817472 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:28.817749 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:28.817817 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:28.821882 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:28.821951 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:28.822011 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:28.827437 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:28.828508 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:28.829550 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:28.839478 3516 server_base.cc:1034] running on GCE node
W20260927 15:02:28.839411 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:28.839397 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:28.839394 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:28.840101 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:28.840834 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:28.842057 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790521348842030 us; error 47 us; skew 500 ppm
Sep 27 15:02:28 dist-test-slave-ml7t krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:28.845305 3516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:28.846371 3516 webserver.cc:460] Webserver started at http://127.0.98.1:37201/ using document root <none> and password file <none>
I20260927 15:02:28.846976 3516 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:28.847040 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:28.847267 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:28.849385 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f19b25870a34836be99f8f864a47eab"
format_stamp: "Formatted at 2026-09-27 15:02:28 on dist-test-slave-ml7t"
server_key: "e893fbc65377dd311c850f10fb8411c3"
server_key_iv: "aaa59f81305410cc94948d9c05d4c43e"
server_key_version: "encryptionkey@0"
I20260927 15:02:28.849998 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f19b25870a34836be99f8f864a47eab"
format_stamp: "Formatted at 2026-09-27 15:02:28 on dist-test-slave-ml7t"
server_key: "e893fbc65377dd311c850f10fb8411c3"
server_key_iv: "aaa59f81305410cc94948d9c05d4c43e"
server_key_version: "encryptionkey@0"
I20260927 15:02:28.853691 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:28.856113 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:28.857429 3516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:28.857573 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "4f19b25870a34836be99f8f864a47eab"
format_stamp: "Formatted at 2026-09-27 15:02:28 on dist-test-slave-ml7t"
server_key: "e893fbc65377dd311c850f10fb8411c3"
server_key_iv: "aaa59f81305410cc94948d9c05d4c43e"
server_key_version: "encryptionkey@0"
I20260927 15:02:28.857683 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:28.879571 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:28.882687 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:28.882851 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:28.883388 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:28.884153 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:28.884198 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:28.884238 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:28.884253 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:28.893153 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34189
I20260927 15:02:28.893179 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34189 every 8 connection(s)
I20260927 15:02:28.894119 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:28.903203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3516
I20260927 15:02:28.903333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:28.903635 392 external_mini_cluster.cc:1428] Setting key c2b9d1ec795df71b36af253ad1ae3be9
Sep 27 15:02:28 dist-test-slave-ml7t krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:28.911057 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:28.895907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52089 (local address 127.0.98.62:46345)
0927 15:02:28.896042 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:28.896046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:28.896818 (+ 772us) server_negotiation.cc:407] Connection header received
0927 15:02:28.898949 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:28.898953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:28.899013 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:28.899122 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:28.900621 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:28.901202 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:28.903149 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:28.903526 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:28.903635 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:28.906929 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:28.906935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:28.906941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:28.906978 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:28.908680 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:28.909227 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:28.909234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:28.909239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:28.909336 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:28.909711 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:28.909718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:28.909722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:28.910022 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:28.910131 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:28.910646 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 15:02:28.910813 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:02:28.911767 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:28.896222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46345 (local address 127.0.98.1:52089)
0927 15:02:28.896642 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:28.896683 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:28.898735 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:28.899303 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:28.899310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:28.899723 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:28.900437 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:28.900460 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:28.901335 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:28.901342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:28.902995 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:28.903009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:28.903710 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:28.903717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:28.903955 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:28.904720 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:28.904736 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:28.906726 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:28.908845 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:28.908849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:28.908862 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:28.909112 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:28.909425 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:28.909427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:28.909429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:28.909580 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:28.910124 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:28.910129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:28.910376 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 15:02:28.910535 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 15:02:28.910837 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260927 15:02:28.913069 3645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46345
I20260927 15:02:28.913309 3645 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:02:28.913883 3645 heartbeater.cc:511] Master 127.0.98.62:46345 requested a full tablet report, sending...
I20260927 15:02:28.915350 3454 ts_manager.cc:194] Registered new tserver with Master: 4f19b25870a34836be99f8f864a47eab (127.0.98.1:34189)
I20260927 15:02:28.916424 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 15:02:29.168584 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.263s user 0.000s sys 0.003s
I20260927 15:02:29.169077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46345
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:29.280229 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:29.280498 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:29.280542 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:29.284392 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:29.284446 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:29.284503 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:29.289497 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:29.290784 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:29.292025 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:29.301570 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:29.301586 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:29.301653 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:29.301836 3650 server_base.cc:1034] running on GCE node
I20260927 15:02:29.302246 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:29.302870 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:29.304040 3650 hybrid_clock.cc:648] HybridClock initialized: now 1790521349304026 us; error 26 us; skew 500 ppm
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:29.306916 3650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:29.307799 3650 webserver.cc:460] Webserver started at http://127.0.98.2:32875/ using document root <none> and password file <none>
I20260927 15:02:29.308328 3650 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:29.308391 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:29.308563 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:29.310741 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "24a605940ca549debb0e055e915590f7"
format_stamp: "Formatted at 2026-09-27 15:02:29 on dist-test-slave-ml7t"
server_key: "c535cd255541d04151afdbbdedfc8fff"
server_key_iv: "7cd9552641959f79b939ba1d24f45d6f"
server_key_version: "encryptionkey@0"
I20260927 15:02:29.311288 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24a605940ca549debb0e055e915590f7"
format_stamp: "Formatted at 2026-09-27 15:02:29 on dist-test-slave-ml7t"
server_key: "c535cd255541d04151afdbbdedfc8fff"
server_key_iv: "7cd9552641959f79b939ba1d24f45d6f"
server_key_version: "encryptionkey@0"
I20260927 15:02:29.315320 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:02:29.318114 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:29.319379 3650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:29.319480 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "24a605940ca549debb0e055e915590f7"
format_stamp: "Formatted at 2026-09-27 15:02:29 on dist-test-slave-ml7t"
server_key: "c535cd255541d04151afdbbdedfc8fff"
server_key_iv: "7cd9552641959f79b939ba1d24f45d6f"
server_key_version: "encryptionkey@0"
I20260927 15:02:29.319568 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:29.343951 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:29.347390 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:29.347541 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:29.348074 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:29.348824 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:29.348878 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:29.348917 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:29.348932 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:29.357828 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46355
I20260927 15:02:29.357868 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46355 every 8 connection(s)
I20260927 15:02:29.358786 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:29.365480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3650
I20260927 15:02:29.365573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:29.365938 392 external_mini_cluster.cc:1428] Setting key ef1fe70f7f6bfa6b7b85f197c7d6a5d5
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:29.373401 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:29.360730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48695 (local address 127.0.98.62:46345)
0927 15:02:29.360865 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:29.360868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:29.361830 (+ 962us) server_negotiation.cc:407] Connection header received
0927 15:02:29.363256 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:29.363259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:29.363313 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:29.363395 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:29.364563 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.365098 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.366520 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.366802 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.366922 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:29.369581 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:29.369585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:29.369587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:29.369619 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:29.371273 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:29.371809 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:29.371814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:29.371815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:29.371906 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:29.372253 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:29.372258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:29.372260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:29.372474 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:29.372594 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:29.373108 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 15:02:29.373230 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:02:29.374332 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:29.361124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46345 (local address 127.0.98.2:48695)
0927 15:02:29.361639 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:29.361691 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:29.363048 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:29.363569 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:29.363575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:29.363951 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:29.364421 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:29.364433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.365223 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.365233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:29.366387 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:29.366409 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.366961 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.366964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:29.367150 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:29.367833 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:29.367848 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:29.369414 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:29.371394 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:29.371419 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:29.371436 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:29.371701 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:29.372010 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:29.372013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:29.372015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:29.372156 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:29.372606 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:29.372614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:29.372863 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 15:02:29.373055 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 15:02:29.373414 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:29.375545 3779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46345
I20260927 15:02:29.375764 3779 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:29.376199 3779 heartbeater.cc:511] Master 127.0.98.62:46345 requested a full tablet report, sending...
I20260927 15:02:29.377315 3454 ts_manager.cc:194] Registered new tserver with Master: 24a605940ca549debb0e055e915590f7 (127.0.98.2:46355)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:29.378005 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:29.393545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46345
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:29.503489 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:29.503736 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:29.503780 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:29.507858 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:29.507920 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:29.508041 3784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:29.513372 3784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:29.514530 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:29.515532 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:29.525466 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:29.525461 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:29.525465 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:29.525749 3784 server_base.cc:1034] running on GCE node
I20260927 15:02:29.526233 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:29.526888 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:29.528062 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790521349528047 us; error 25 us; skew 500 ppm
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:29.530936 3784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:29.531865 3784 webserver.cc:460] Webserver started at http://127.0.98.3:38241/ using document root <none> and password file <none>
I20260927 15:02:29.532506 3784 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:29.532569 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:29.532733 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:29.534799 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ac1f450e63f42eebfc4b493af3f418d"
format_stamp: "Formatted at 2026-09-27 15:02:29 on dist-test-slave-ml7t"
server_key: "203b63600869611c0d56b9302689217d"
server_key_iv: "7c8d1e437e30ba1e31fc4dcb659e87ed"
server_key_version: "encryptionkey@0"
I20260927 15:02:29.535302 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ac1f450e63f42eebfc4b493af3f418d"
format_stamp: "Formatted at 2026-09-27 15:02:29 on dist-test-slave-ml7t"
server_key: "203b63600869611c0d56b9302689217d"
server_key_iv: "7c8d1e437e30ba1e31fc4dcb659e87ed"
server_key_version: "encryptionkey@0"
I20260927 15:02:29.538910 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:02:29.541280 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:29.542563 3784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:29.542727 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "4ac1f450e63f42eebfc4b493af3f418d"
format_stamp: "Formatted at 2026-09-27 15:02:29 on dist-test-slave-ml7t"
server_key: "203b63600869611c0d56b9302689217d"
server_key_iv: "7c8d1e437e30ba1e31fc4dcb659e87ed"
server_key_version: "encryptionkey@0"
I20260927 15:02:29.542868 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:29.562593 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:29.566128 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:29.566349 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:29.566960 3784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:29.567818 3784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:29.567891 3784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:29.567963 3784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:29.568017 3784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:29.577603 3784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33291
I20260927 15:02:29.577625 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33291 every 8 connection(s)
I20260927 15:02:29.578575 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:29.579505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3784
I20260927 15:02:29.579625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:29.579921 392 external_mini_cluster.cc:1428] Setting key 0a11494a22434b36277c931a0ca30b57
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:29.595297 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:29.581300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59047 (local address 127.0.98.62:46345)
0927 15:02:29.581428 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:29.581433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:29.582200 (+ 767us) server_negotiation.cc:407] Connection header received
0927 15:02:29.583548 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:29.583551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:29.583593 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:29.583660 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:29.584742 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.585280 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.587102 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.587506 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.587610 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:29.590577 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:29.590582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:29.590585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:29.590618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:29.592680 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:29.593352 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:29.593356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:29.593357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:29.593449 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:29.593963 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:29.593967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:29.593968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:29.594212 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:29.594341 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:29.595017 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 15:02:29.595140 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:02:29.596230 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:29.580876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46345 (local address 127.0.98.3:59047)
0927 15:02:29.581434 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:29.582097 (+ 663us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:29.583357 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:29.583799 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:29.583806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:29.584156 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:29.584618 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:29.584630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.585490 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.585496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:29.586942 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:29.586955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.587663 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.587667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:29.587851 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:29.588480 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:29.588496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:29.590354 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:29.592829 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:29.592836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:29.592853 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:29.593216 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:29.593577 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:29.593580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:29.593582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:29.593848 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:29.594408 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:29.594415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:29.594761 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 15:02:29.594964 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 15:02:29.595316 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":128,"threads_started":1}
I20260927 15:02:29.597376 3913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46345
I20260927 15:02:29.597625 3913 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:29.598112 3913 heartbeater.cc:511] Master 127.0.98.62:46345 requested a full tablet report, sending...
I20260927 15:02:29.599113 3454 ts_manager.cc:194] Registered new tserver with Master: 4ac1f450e63f42eebfc4b493af3f418d (127.0.98.3:33291)
I20260927 15:02:29.599834 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59047
I20260927 15:02:29.606336 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521349, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521349, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:29.632246 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:29.623675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.0.98.62:46345)
0927 15:02:29.623829 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:29.623832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:29.623972 (+ 140us) server_negotiation.cc:407] Connection header received
0927 15:02:29.624102 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:29.624105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:29.624149 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:29.624214 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:29.624931 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.625464 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.626285 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:29.626499 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:29.626582 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:29.629084 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:29.629087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:29.629089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:29.629113 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:29.630588 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:29.631103 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:29.631106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:29.631107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:29.631182 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:29.631462 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:29.631465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:29.631466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:29.631700 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:29.631812 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:29.632026 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 15:02:29.632126 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 15:02:29.632686 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33586
I20260927 15:02:29.633625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3516
I20260927 15:02:29.639745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3650
I20260927 15:02:29.644958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3784
I20260927 15:02:29.650269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3422
2026-09-27T15:02:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2407 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3925](info): set up 2 sockets
Sep 27 15:02:29 dist-test-slave-ml7t krb5kdc[3925](info): commencing operation
krb5kdc: starting...
W20260927 15:02:30.689900 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:30 dist-test-slave-ml7t krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:30.810968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44925
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:30.920192 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:30.920526 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:30.920606 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:30.924615 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:30.924727 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:30.924758 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:30.924782 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:30.924808 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:30.930199 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:30.931353 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:30.932412 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:30.941222 3938 server_base.cc:1034] running on GCE node
W20260927 15:02:30.941330 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:30.941176 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:30.941176 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:30.942087 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:30.943243 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:30.944446 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790521350944417 us; error 45 us; skew 500 ppm
Sep 27 15:02:30 dist-test-slave-ml7t krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:30.951617 3938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:30.952597 3938 webserver.cc:460] Webserver started at http://127.0.98.62:43097/ using document root <none> and password file <none>
I20260927 15:02:30.953166 3938 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:30.953227 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:30.953393 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:30.955440 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "722e6e2bb90b413bbea5159873dc6632"
format_stamp: "Formatted at 2026-09-27 15:02:30 on dist-test-slave-ml7t"
server_key: "cb7ebfc61746097f003fb7651f036f40"
server_key_iv: "00be7da0c676e822e21ee2da1ede0de1"
server_key_version: "encryptionkey@0"
I20260927 15:02:30.955945 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "722e6e2bb90b413bbea5159873dc6632"
format_stamp: "Formatted at 2026-09-27 15:02:30 on dist-test-slave-ml7t"
server_key: "cb7ebfc61746097f003fb7651f036f40"
server_key_iv: "00be7da0c676e822e21ee2da1ede0de1"
server_key_version: "encryptionkey@0"
I20260927 15:02:30.959599 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 15:02:30.961982 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:30.963315 3938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:30.963444 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "722e6e2bb90b413bbea5159873dc6632"
format_stamp: "Formatted at 2026-09-27 15:02:30 on dist-test-slave-ml7t"
server_key: "cb7ebfc61746097f003fb7651f036f40"
server_key_iv: "00be7da0c676e822e21ee2da1ede0de1"
server_key_version: "encryptionkey@0"
I20260927 15:02:30.963548 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:30.972503 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:30.975662 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:30.975915 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:30.982679 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44925
I20260927 15:02:30.982707 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44925 every 8 connection(s)
I20260927 15:02:30.983639 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:30.986601 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:30.987465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 3938
I20260927 15:02:30.987596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:30.987922 392 external_mini_cluster.cc:1428] Setting key e15495ec3d6c23552a159d4f3529456a
I20260927 15:02:30.992900 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632: Bootstrap starting.
I20260927 15:02:30.995340 4006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:30.996091 4006 log.cc:824] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:02:30 dist-test-slave-ml7t krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:30.998260 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632: No bootstrap required, opened a new log
I20260927 15:02:31.002091 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722e6e2bb90b413bbea5159873dc6632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260927 15:02:31.002367 4006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:31.002453 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722e6e2bb90b413bbea5159873dc6632, State: Initialized, Role: FOLLOWER
I20260927 15:02:31.002959 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722e6e2bb90b413bbea5159873dc6632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260927 15:02:31.003116 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:31.003202 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:31.003289 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:30.989513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.0.98.62:44925)
0927 15:02:30.990013 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:30.990027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:30.990061 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:02:30.990655 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:30.990672 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:30.991026 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:30.991347 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:30.992148 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:30.993146 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:30.994047 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:30.994506 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:30.994671 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:30.997390 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:30.997398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:30.997418 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:30.997443 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:30.999687 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.000331 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.000336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.000342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.000475 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.000805 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.000812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.000815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.001242 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:31.001478 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:31.001825 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 15:02:31.002304 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":159,"threads_started":1}
I20260927 15:02:31.003305 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:31.004982 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722e6e2bb90b413bbea5159873dc6632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260927 15:02:31.005374 4006 leader_election.cc:302] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 722e6e2bb90b413bbea5159873dc6632; no voters:
I20260927 15:02:31.005751 4006 leader_election.cc:288] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:31.005860 4011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:31.006035 4011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [term 1 LEADER]: Becoming Leader. State: Replica: 722e6e2bb90b413bbea5159873dc6632, State: Running, Role: LEADER
I20260927 15:02:31.006421 4011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722e6e2bb90b413bbea5159873dc6632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260927 15:02:31.007141 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:31.008203 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722e6e2bb90b413bbea5159873dc6632. Latest consensus state: current_term: 1 leader_uuid: "722e6e2bb90b413bbea5159873dc6632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722e6e2bb90b413bbea5159873dc6632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } } }
I20260927 15:02:31.008631 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:31.008523 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "722e6e2bb90b413bbea5159873dc6632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722e6e2bb90b413bbea5159873dc6632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } } }
I20260927 15:02:31.009002 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632 [sys.catalog]: This master's current role is: LEADER
W20260927 15:02:31.012116 4026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:02:31.012200 4026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:02:31.012279 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:31.012946 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:31.017992 4022 catalog_manager.cc:1347] Generated new cluster ID: a21689389e7d4310be55f986f2a1e93b
I20260927 15:02:31.018060 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:31.027799 4022 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:31.029350 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:31.037416 4022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 722e6e2bb90b413bbea5159873dc6632: Generated new TSK 0
I20260927 15:02:31.038358 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:31.068464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:31.178754 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:31.179041 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:31.179116 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:31.183109 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:31.183202 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:31.183308 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:31.188479 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:31.189558 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:31.190640 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:31.205965 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:31.205929 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:31.205971 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:31.209585 4032 server_base.cc:1034] running on GCE node
I20260927 15:02:31.210144 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:31.211016 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:31.212203 4032 hybrid_clock.cc:648] HybridClock initialized: now 1790521351212176 us; error 72 us; skew 500 ppm
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:31.216572 4032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:31.217922 4032 webserver.cc:460] Webserver started at http://127.0.98.1:46385/ using document root <none> and password file <none>
I20260927 15:02:31.218653 4032 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:31.218743 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:31.219017 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:31.222267 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "4449ca2076a543cd86e3e1cdf7b08b84"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "e50e2127922ffd102422f4ef24990b4c"
server_key_iv: "bb5417ab62012a5fa343185c8eddabbb"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.222971 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4449ca2076a543cd86e3e1cdf7b08b84"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "e50e2127922ffd102422f4ef24990b4c"
server_key_iv: "bb5417ab62012a5fa343185c8eddabbb"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.227435 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:02:31.230472 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.232124 4032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:31.232275 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "4449ca2076a543cd86e3e1cdf7b08b84"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "e50e2127922ffd102422f4ef24990b4c"
server_key_iv: "bb5417ab62012a5fa343185c8eddabbb"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.232414 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:31.258810 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:31.261763 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:31.262020 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:31.262625 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:31.263597 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:31.263664 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.263738 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:31.263789 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.272933 4032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33817
I20260927 15:02:31.272956 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33817 every 8 connection(s)
I20260927 15:02:31.273998 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:31.275525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4032
I20260927 15:02:31.275671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:31.276023 392 external_mini_cluster.cc:1428] Setting key cf240b0db805d73a0e08dec50eb32166
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:31.290880 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.275936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41385 (local address 127.0.98.62:44925)
0927 15:02:31.276100 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:31.276107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:31.277207 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 15:02:31.279033 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:31.279039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:31.279105 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:31.279199 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:31.280840 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.281827 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.283586 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.283820 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.283902 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.286946 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:31.286949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:31.286952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:31.286977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:31.288455 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.289056 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.289060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.289061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.289142 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.289556 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.289560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.289562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.289918 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:31.290028 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:31.290563 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 15:02:31.290699 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:02:31.291760 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.276440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44925 (local address 127.0.98.1:41385)
0927 15:02:31.277032 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:31.277090 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:31.278743 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:31.279424 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:31.279436 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:31.280035 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:31.280659 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:31.280674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.281963 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.281966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:31.283430 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:31.283445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.283982 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.283986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:31.284216 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.284896 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:31.284919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:31.286763 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:31.288580 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:31.288586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:31.288600 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:31.288960 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:31.289231 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:31.289233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:31.289235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:31.289433 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:31.290026 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:31.290033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:31.290311 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 15:02:31.290513 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 15:02:31.290873 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":143,"threads_started":1}
I20260927 15:02:31.293100 4161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44925
I20260927 15:02:31.293377 4161 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:31.294059 4161 heartbeater.cc:511] Master 127.0.98.62:44925 requested a full tablet report, sending...
I20260927 15:02:31.295748 3970 ts_manager.cc:194] Registered new tserver with Master: 4449ca2076a543cd86e3e1cdf7b08b84 (127.0.98.1:33817)
I20260927 15:02:31.296883 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:31.305125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:31.418081 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:31.418354 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:31.418398 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:31.422374 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:31.422432 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:31.422492 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:31.427737 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:31.428782 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:31.429827 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:31.439430 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:31.439467 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:31.439553 4166 server_base.cc:1034] running on GCE node
W20260927 15:02:31.439388 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:31.440085 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:31.440850 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:31.442006 4166 hybrid_clock.cc:648] HybridClock initialized: now 1790521351441962 us; error 59 us; skew 500 ppm
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:31.444715 4166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:31.445602 4166 webserver.cc:460] Webserver started at http://127.0.98.2:35545/ using document root <none> and password file <none>
I20260927 15:02:31.446211 4166 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:31.446323 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:31.446552 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:31.448545 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "67fca7a608244841818a522e9f938b60"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "bec0b81ffe78c4f40f73fab38b2fb44a"
server_key_iv: "7c6f236ab988138942a7c2d6e5c27f55"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.449113 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "67fca7a608244841818a522e9f938b60"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "bec0b81ffe78c4f40f73fab38b2fb44a"
server_key_iv: "7c6f236ab988138942a7c2d6e5c27f55"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.453053 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:02:31.455714 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.457103 4166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:31.457267 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "67fca7a608244841818a522e9f938b60"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "bec0b81ffe78c4f40f73fab38b2fb44a"
server_key_iv: "7c6f236ab988138942a7c2d6e5c27f55"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.457396 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:31.475301 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:31.478821 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:31.479036 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:31.479692 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:31.480567 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:31.480639 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.480706 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:31.480756 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.489825 4166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33223
I20260927 15:02:31.489886 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33223 every 8 connection(s)
I20260927 15:02:31.490852 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:31.491699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4166
I20260927 15:02:31.491830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:31.492139 392 external_mini_cluster.cc:1428] Setting key 94ea9235d452eede2559d099a1059e60
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:31.506541 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.492935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46181 (local address 127.0.98.62:44925)
0927 15:02:31.493101 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:31.493105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:31.494115 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 15:02:31.496087 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:31.496094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:31.496157 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:31.496261 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:31.497459 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.498445 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.499596 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.500015 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.500145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.502919 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:31.502922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:31.502924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:31.502949 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:31.504313 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.504949 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.504954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.504957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.505047 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.505413 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.505418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.505420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.505612 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:31.505749 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:31.506249 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 15:02:31.506377 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:02:31.507467 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.493377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44925 (local address 127.0.98.2:46181)
0927 15:02:31.493938 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:31.493999 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:31.495863 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:31.496446 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:31.496453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:31.496817 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:31.497312 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:31.497325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.498573 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.498575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:31.499466 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:31.499475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.500163 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.500165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:31.500329 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.500979 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:31.500995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:31.502758 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:31.504432 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:31.504437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:31.504449 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:31.504849 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:31.505142 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:31.505145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:31.505146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:31.505298 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:31.505727 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:31.505733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:31.506014 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 15:02:31.506209 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 15:02:31.506540 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":150,"threads_started":1}
I20260927 15:02:31.508814 4295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44925
I20260927 15:02:31.509105 4295 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:31.509648 4295 heartbeater.cc:511] Master 127.0.98.62:44925 requested a full tablet report, sending...
I20260927 15:02:31.510763 3970 ts_manager.cc:194] Registered new tserver with Master: 67fca7a608244841818a522e9f938b60 (127.0.98.2:33223)
I20260927 15:02:31.511591 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:31.521512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:31.631573 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:31.631917 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:31.631969 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:31.636425 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:31.636502 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:31.636601 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:31.642160 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:31.643239 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:31.644273 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:31.654199 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:31.654188 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:31.654361 4300 server_base.cc:1034] running on GCE node
W20260927 15:02:31.654188 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:31.654935 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:31.655619 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:31.656827 4300 hybrid_clock.cc:648] HybridClock initialized: now 1790521351656809 us; error 27 us; skew 500 ppm
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:31.659957 4300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:31.660918 4300 webserver.cc:460] Webserver started at http://127.0.98.3:44115/ using document root <none> and password file <none>
I20260927 15:02:31.661455 4300 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:31.661514 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:31.661748 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:31.664024 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b3fda8be5c64334bcc1f2ab8803a91f"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "1d8d9f42ebea64de00ddd8aaa92ee4db"
server_key_iv: "8355ab765c60933e76b8ef29f97aa267"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.664562 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b3fda8be5c64334bcc1f2ab8803a91f"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "1d8d9f42ebea64de00ddd8aaa92ee4db"
server_key_iv: "8355ab765c60933e76b8ef29f97aa267"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.668728 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:02:31.671214 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.672637 4300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:31.672749 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "0b3fda8be5c64334bcc1f2ab8803a91f"
format_stamp: "Formatted at 2026-09-27 15:02:31 on dist-test-slave-ml7t"
server_key: "1d8d9f42ebea64de00ddd8aaa92ee4db"
server_key_iv: "8355ab765c60933e76b8ef29f97aa267"
server_key_version: "encryptionkey@0"
I20260927 15:02:31.672842 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:31.682893 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:31.686534 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:31.686700 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:31.687245 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:31.688081 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:31.688131 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.688181 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:31.688206 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:31.697440 4300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42109
I20260927 15:02:31.697518 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42109 every 8 connection(s)
I20260927 15:02:31.698467 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:31.707701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4300
I20260927 15:02:31.707850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:31.708192 392 external_mini_cluster.cc:1428] Setting key 37a7b568c1c04ef42af7f2808304cef1
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:31.712487 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.700359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40491 (local address 127.0.98.62:44925)
0927 15:02:31.700506 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:31.700509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:31.701385 (+ 876us) server_negotiation.cc:407] Connection header received
0927 15:02:31.702863 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:31.702867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:31.702925 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:31.703031 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:31.704162 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.704697 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.705883 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.706114 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.706226 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.708934 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:31.708938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:31.708941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:31.708967 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:31.710503 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.711017 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.711022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.711023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.711106 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.711453 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.711458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.711460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.711644 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:31.711761 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:31.712214 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 15:02:31.712333 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:02:31.713267 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.700786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44925 (local address 127.0.98.3:40491)
0927 15:02:31.701217 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:31.701271 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:31.702621 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:31.703202 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:31.703209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:31.703562 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:31.704019 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:31.704033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.704822 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.704828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:31.705723 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:31.705732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.706269 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.706274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:31.706442 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.707029 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:31.707045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:31.708771 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:31.710636 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:31.710642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:31.710657 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:31.710913 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:31.711205 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:31.711208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:31.711210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:31.711351 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:31.711748 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:31.711755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:31.711991 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 15:02:31.712175 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 15:02:31.712491 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260927 15:02:31.714324 4429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44925
I20260927 15:02:31.714532 4429 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:31.714952 4429 heartbeater.cc:511] Master 127.0.98.62:44925 requested a full tablet report, sending...
I20260927 15:02:31.715950 3970 ts_manager.cc:194] Registered new tserver with Master: 0b3fda8be5c64334bcc1f2ab8803a91f (127.0.98.3:42109)
I20260927 15:02:31.716550 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40491
I20260927 15:02:31.722069 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:31.748368 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.738352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.0.98.1:33817)
0927 15:02:31.738653 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:31.738670 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:31.738697 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:02:31.738784 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:31.738797 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:31.739019 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:31.739179 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:31.739878 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.740638 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.741485 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.741976 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.742083 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.744538 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:31.744546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:31.744561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:31.744586 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:31.746460 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.746948 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.746961 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.746962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.747042 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.747326 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.747329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.747331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.747587 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:31.747850 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:31.748016 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 15:02:31.748177 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:31.759938 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:31.751459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.98.62:44925)
0927 15:02:31.751581 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:31.751585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:31.751712 (+ 127us) server_negotiation.cc:407] Connection header received
0927 15:02:31.751814 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:31.751816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:31.751856 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:31.751924 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:31.752532 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.753070 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.753960 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:31.754170 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:31.754243 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:31.756751 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:31.756753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:31.756755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:31.756778 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:31.758280 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.758695 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.758701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.758704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.758784 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:31.759051 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:31.759056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:31.759058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:31.759277 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:31.759416 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:31.759645 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 15:02:31.759775 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 15:02:31.760346 3970 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508
I20260927 15:02:31.761114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4032
I20260927 15:02:31.767901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4166
I20260927 15:02:31.773342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4300
I20260927 15:02:31.779076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 3938
2026-09-27T15:02:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2128 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[4441](info): set up 2 sockets
Sep 27 15:02:31 dist-test-slave-ml7t krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260927 15:02:32.918133 392 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:32 dist-test-slave-ml7t krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:33.041086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39537
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:33.149744 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:33.150126 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:33.150218 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:33.154335 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:33.154415 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:33.154452 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:33.154511 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:33.154551 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:33.159720 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:33.160801 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:33.161871 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:33.170130 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:33.170130 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:33.170204 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:33.170507 4454 server_base.cc:1034] running on GCE node
I20260927 15:02:33.171238 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:33.172304 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:33.173489 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790521353173479 us; error 27 us; skew 500 ppm
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:33.176236 4454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:33.176976 4454 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:33.177066 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:33.177294 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:33.179375 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "15edc5715b7b4c898edb7c126c50455c"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "1d1307d788588d794da2b6cad6a65a6d"
server_key_iv: "73816ac1693b40e7203830e9f10247d1"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.179931 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "15edc5715b7b4c898edb7c126c50455c"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "1d1307d788588d794da2b6cad6a65a6d"
server_key_iv: "73816ac1693b40e7203830e9f10247d1"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.183548 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 15:02:33.185927 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.187179 4454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:33.187322 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "15edc5715b7b4c898edb7c126c50455c"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "1d1307d788588d794da2b6cad6a65a6d"
server_key_iv: "73816ac1693b40e7203830e9f10247d1"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.187428 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:33.197659 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:33.200306 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:33.200523 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:33.207261 4454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39537
I20260927 15:02:33.207523 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39537 every 8 connection(s)
I20260927 15:02:33.207901 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:33.210420 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:33.215559 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c: Bootstrap starting.
I20260927 15:02:33.216886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4454
I20260927 15:02:33.217026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:33.217289 392 external_mini_cluster.cc:1428] Setting key 37392dfda272a75367889ce0fc8c7047
I20260927 15:02:33.218040 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:33.218779 4521 log.cc:824] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:33.220736 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c: No bootstrap required, opened a new log
I20260927 15:02:33.224880 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15edc5715b7b4c898edb7c126c50455c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 15:02:33.225180 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:33.225236 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15edc5715b7b4c898edb7c126c50455c, State: Initialized, Role: FOLLOWER
I20260927 15:02:33.225693 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15edc5715b7b4c898edb7c126c50455c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 15:02:33.225868 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:33.225950 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:33.226048 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 0 FOLLOWER]: Advancing to term 1
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:33.226879 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15edc5715b7b4c898edb7c126c50455c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 15:02:33.227201 4521 leader_election.cc:302] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15edc5715b7b4c898edb7c126c50455c; no voters:
I20260927 15:02:33.227516 4521 leader_election.cc:288] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:33.227612 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:33.227833 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [term 1 LEADER]: Becoming Leader. State: Replica: 15edc5715b7b4c898edb7c126c50455c, State: Running, Role: LEADER
I20260927 15:02:33.228123 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15edc5715b7b4c898edb7c126c50455c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 15:02:33.228811 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:33.229153 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15edc5715b7b4c898edb7c126c50455c. Latest consensus state: current_term: 1 leader_uuid: "15edc5715b7b4c898edb7c126c50455c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15edc5715b7b4c898edb7c126c50455c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } } }
I20260927 15:02:33.229264 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:33.229610 4530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:33.230988 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15edc5715b7b4c898edb7c126c50455c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15edc5715b7b4c898edb7c126c50455c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } } }
I20260927 15:02:33.231086 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:33.232381 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.218353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.0.98.62:39537)
0927 15:02:33.218796 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.218811 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.218844 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:02:33.219637 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.219655 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.219967 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.220228 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:33.221109 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.222091 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.223276 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.223705 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.223828 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.226738 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:33.226744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:33.226766 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:33.226811 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:33.228839 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.229357 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.229363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.229369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.229520 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.229937 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.229941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.229942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.230484 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:33.230676 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.231164 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.231497 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":291,"thread_start_us":155,"threads_started":1}
I20260927 15:02:33.233196 4530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:33.238641 4530 catalog_manager.cc:1347] Generated new cluster ID: 4df6655dee9a457b9251280f99c0e5f6
I20260927 15:02:33.238714 4530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:33.265040 4530 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:33.266425 4530 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:33.273648 4530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c: Generated new TSK 0
I20260927 15:02:33.274511 4530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:33.304541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:33.419191 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:33.419483 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:33.419530 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:33.423794 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:33.423862 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:33.423928 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:33.429376 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:33.430554 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:33.431599 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:33.440963 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:33.440963 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:33.441220 4547 server_base.cc:1034] running on GCE node
W20260927 15:02:33.441037 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:33.441847 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:33.442507 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:33.443696 4547 hybrid_clock.cc:648] HybridClock initialized: now 1790521353443668 us; error 39 us; skew 500 ppm
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:33.446571 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:33.447263 4547 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:33.447328 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:33.447492 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:33.449517 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "44bbf996825448e890270b05f37bf139"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "a6f420c56d27278ef324a5063b595e12"
server_key_iv: "f1c8697790fae5dab8c1199007ab3ce1"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.450137 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "44bbf996825448e890270b05f37bf139"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "a6f420c56d27278ef324a5063b595e12"
server_key_iv: "f1c8697790fae5dab8c1199007ab3ce1"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.453905 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:02:33.456310 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.457614 4547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:33.457710 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "44bbf996825448e890270b05f37bf139"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "a6f420c56d27278ef324a5063b595e12"
server_key_iv: "f1c8697790fae5dab8c1199007ab3ce1"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.457867 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:33.492139 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:33.495421 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:33.495579 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:33.496155 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:33.496953 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:33.497005 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.497045 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:33.497061 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.506126 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35261
I20260927 15:02:33.506206 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35261 every 8 connection(s)
I20260927 15:02:33.506742 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:33.511871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4547
I20260927 15:02:33.512005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:33.512317 392 external_mini_cluster.cc:1428] Setting key 8cde0aef470d0da4d90e8f2c11737438
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:33.520862 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.508439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56877 (local address 127.0.98.62:39537)
0927 15:02:33.508588 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.508592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.509388 (+ 796us) server_negotiation.cc:407] Connection header received
0927 15:02:33.510747 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.510750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.510800 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.510874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:33.512038 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.512766 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.514179 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.514415 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.514527 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.516921 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:33.516924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:33.516927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:33.516949 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:33.518466 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.519028 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.519033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.519035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.519118 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.519556 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.519560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.519562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.519793 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:33.519911 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.520545 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.520681 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:02:33.521926 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.508800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39537 (local address 127.0.98.1:56877)
0927 15:02:33.509233 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:33.509275 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:33.510559 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:33.511025 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:33.511032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:33.511391 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:33.511851 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.511864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.512900 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.512904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.514055 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.514065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.514559 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.514563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.514727 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.515337 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:33.515352 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:33.516739 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:33.518597 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:33.518603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:33.518619 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:33.518905 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:33.519222 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:33.519225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:33.519236 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:33.519406 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:33.519910 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:33.519917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:33.520230 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 15:02:33.520492 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 15:02:33.520864 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260927 15:02:33.523219 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39537
I20260927 15:02:33.523497 4675 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:33.524048 4675 heartbeater.cc:511] Master 127.0.98.62:39537 requested a full tablet report, sending...
I20260927 15:02:33.525637 4485 ts_manager.cc:194] Registered new tserver with Master: 44bbf996825448e890270b05f37bf139 (127.0.98.1:35261)
I20260927 15:02:33.526857 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:33.541218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:33.654444 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:33.654718 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:33.654762 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:33.658808 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:33.658861 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:33.658919 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:33.664191 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:33.665185 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:33.666275 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:33.676254 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:33.676259 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:33.676385 4680 server_base.cc:1034] running on GCE node
W20260927 15:02:33.676255 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:33.676987 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:33.677675 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:33.678874 4680 hybrid_clock.cc:648] HybridClock initialized: now 1790521353678842 us; error 57 us; skew 500 ppm
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:33.681648 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:33.682463 4680 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:33.682585 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:33.682828 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:33.684871 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "151c7f3b80f94e1994833f7b9426bd02"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "e87c046ed50cff6c036ce1d8feaf18b2"
server_key_iv: "7fd8fe7192f7f799a8048ff76073d1ef"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.685529 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "151c7f3b80f94e1994833f7b9426bd02"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "e87c046ed50cff6c036ce1d8feaf18b2"
server_key_iv: "7fd8fe7192f7f799a8048ff76073d1ef"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.689268 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:33.691845 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.693076 4680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:33.693208 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "151c7f3b80f94e1994833f7b9426bd02"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "e87c046ed50cff6c036ce1d8feaf18b2"
server_key_iv: "7fd8fe7192f7f799a8048ff76073d1ef"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.693338 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:33.703254 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:33.706120 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:33.706311 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:33.706842 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:33.707612 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:33.707683 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.707758 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:33.707808 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.716361 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42569
I20260927 15:02:33.716378 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42569 every 8 connection(s)
I20260927 15:02:33.716928 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:33.717569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4680
I20260927 15:02:33.717694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:33.718078 392 external_mini_cluster.cc:1428] Setting key c2562e44ff26d5462946cbf2d4853298
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:33.730608 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.718695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41363 (local address 127.0.98.62:39537)
0927 15:02:33.718830 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.718835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.719694 (+ 859us) server_negotiation.cc:407] Connection header received
0927 15:02:33.721103 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.721106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.721159 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.721237 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:33.722440 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.722964 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.724077 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.724296 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.724410 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.726878 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:33.726885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:33.726890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:33.726923 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:33.728258 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.728909 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.728915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.728920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.729038 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.729417 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.729422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.729424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.729606 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:33.729714 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.730324 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.730440 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:33.731547 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.719070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39537 (local address 127.0.98.2:41363)
0927 15:02:33.719545 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:33.719595 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:33.720896 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:33.721405 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:33.721411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:33.721751 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:33.722297 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.722311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.723097 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.723100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.723953 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.723961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.724442 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.724446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.724608 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.725189 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:33.725204 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:33.726672 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:33.728428 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:33.728438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:33.728449 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:33.728722 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:33.729131 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:33.729135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:33.729137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:33.729322 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:33.729711 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:33.729718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:33.730081 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 15:02:33.730275 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 15:02:33.730589 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":140,"threads_started":1}
I20260927 15:02:33.732853 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39537
I20260927 15:02:33.733155 4808 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:33.733700 4808 heartbeater.cc:511] Master 127.0.98.62:39537 requested a full tablet report, sending...
I20260927 15:02:33.734850 4485 ts_manager.cc:194] Registered new tserver with Master: 151c7f3b80f94e1994833f7b9426bd02 (127.0.98.2:42569)
I20260927 15:02:33.735422 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:33.748580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:33.860801 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:33.861083 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:33.861163 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:33.865322 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:33.865401 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:33.865494 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:33.871052 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:33.872154 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:33.873196 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:33.883181 4813 server_base.cc:1034] running on GCE node
W20260927 15:02:33.883152 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:33.883147 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:33.883148 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:33.883865 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:33.884553 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:33.885713 4813 hybrid_clock.cc:648] HybridClock initialized: now 1790521353885698 us; error 32 us; skew 500 ppm
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:33.888563 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:33.889320 4813 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:33.889416 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:33.889664 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:33.891919 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef2366e9f71a4bf2b0ef53733971a801"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "cb11dbbaf3c4e78fac85ed7452ede635"
server_key_iv: "6a05a6180cf3e8228f733ac4102b5885"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.892485 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef2366e9f71a4bf2b0ef53733971a801"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "cb11dbbaf3c4e78fac85ed7452ede635"
server_key_iv: "6a05a6180cf3e8228f733ac4102b5885"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.896122 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 15:02:33.898531 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.899674 4813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:33.899811 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "ef2366e9f71a4bf2b0ef53733971a801"
format_stamp: "Formatted at 2026-09-27 15:02:33 on dist-test-slave-ml7t"
server_key: "cb11dbbaf3c4e78fac85ed7452ede635"
server_key_iv: "6a05a6180cf3e8228f733ac4102b5885"
server_key_version: "encryptionkey@0"
I20260927 15:02:33.899917 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:33.910247 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:33.912936 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:33.913128 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:33.913679 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:33.914502 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:33.914573 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.914638 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:33.914690 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:33.923025 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42693
I20260927 15:02:33.923043 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42693 every 8 connection(s)
I20260927 15:02:33.923605 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:33.924909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4813
I20260927 15:02:33.925061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:33.925436 392 external_mini_cluster.cc:1428] Setting key e13bf190d9eecda586afc75e78c7cc1f
Sep 27 15:02:33 dist-test-slave-ml7t krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:33.937372 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.925318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36677 (local address 127.0.98.62:39537)
0927 15:02:33.925857 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.925861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.926549 (+ 688us) server_negotiation.cc:407] Connection header received
0927 15:02:33.927980 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.927983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.928032 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.928101 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:33.929308 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.929975 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.931127 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.931353 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.931436 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.933937 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:33.933940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:33.933942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:33.933968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:33.935267 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.935775 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.935778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.935780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.935865 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.936230 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.936233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.936235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.936425 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:33.936510 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.937126 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.937245 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
I20260927 15:02:33.938544 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.925770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39537 (local address 127.0.98.3:36677)
0927 15:02:33.926392 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:33.926447 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:33.927801 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:33.928258 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:33.928315 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:33.928689 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:33.929154 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.929167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.930095 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.930098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.931014 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.931023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.931481 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.931484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.931619 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.932198 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:33.932213 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:33.933704 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:33.935380 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:33.935385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:33.935397 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:33.935655 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:33.935959 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:33.935962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:33.935963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:33.936107 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:33.936574 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:33.936580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:33.936878 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 15:02:33.937069 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 15:02:33.937368 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":163,"threads_started":1}
I20260927 15:02:33.939622 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39537
I20260927 15:02:33.939867 4941 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:33.940310 4941 heartbeater.cc:511] Master 127.0.98.62:39537 requested a full tablet report, sending...
I20260927 15:02:33.941493 4485 ts_manager.cc:194] Registered new tserver with Master: ef2366e9f71a4bf2b0ef53733971a801 (127.0.98.3:42693)
I20260927 15:02:33.942099 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36677
I20260927 15:02:33.951090 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:02:33.960041 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.952890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.0.98.62:39537)
0927 15:02:33.953009 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.953013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.953207 (+ 194us) server_negotiation.cc:407] Connection header received
0927 15:02:33.953330 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.953332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.953369 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.953432 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:33.954233 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.954719 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.955526 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.955721 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.955809 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.957270 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:33.957273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:33.957275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:33.957295 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:33.958653 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.959062 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.959064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.959065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.959131 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:33.959390 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:33.959392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:33.959393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:33.959570 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:33.959665 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.959854 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.959934 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:02:33.962858 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:33.965761 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:33.980118 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.974134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42693 (local address 127.0.0.1:46852)
0927 15:02:33.974921 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:33.974969 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:33.975100 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:33.975728 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:33.975735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:33.975764 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:33.975927 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.975935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.977086 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.977092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.978392 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.978405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.979558 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.979561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.979669 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.979705 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:02:33.979839 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:02:33.979955 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":43,"spinlock_wait_cycles":1280,"thread_start_us":75,"threads_started":1}
I20260927 15:02:33.980127 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.974245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.0.98.3:42693)
0927 15:02:33.974625 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.974636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.975256 (+ 620us) server_negotiation.cc:407] Connection header received
0927 15:02:33.975341 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.975347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.975550 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.975704 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:33.976045 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.976963 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.978542 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.979435 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.979560 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.979704 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.979866 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.979955 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":85,"threads_started":1}
I20260927 15:02:33.980599 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.974239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35261 (local address 127.0.0.1:33966)
0927 15:02:33.975101 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:33.975113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:33.975251 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:33.975906 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:33.975908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:33.975922 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:33.976026 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.976032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.977495 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.977499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.979028 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.979038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.980179 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.980182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.980327 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.980347 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:02:33.980409 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:02:33.980468 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":71,"thread_start_us":100,"threads_started":1}
I20260927 15:02:33.981089 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.974613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.98.1:35261)
0927 15:02:33.974973 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.974980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.975186 (+ 206us) server_negotiation.cc:407] Connection header received
0927 15:02:33.975383 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.975387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.975550 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.975704 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:33.976137 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.977351 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.979149 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.979886 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.979971 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.980097 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.980843 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.980943 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":102,"threads_started":1}
I20260927 15:02:33.981084 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.974830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42569 (local address 127.0.0.1:36558)
0927 15:02:33.975715 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:33.975727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:33.975824 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:33.976347 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:33.976350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:33.976368 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:33.976446 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.976451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.977818 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.977822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.978963 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:33.978973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.980161 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.980165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:33.980579 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.980605 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:02:33.980888 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 15:02:33.980952 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":1,"spinlock_wait_cycles":3072,"thread_start_us":84,"threads_started":1}
I20260927 15:02:33.981171 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:33.975214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.0.98.2:42569)
0927 15:02:33.975560 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:33.975569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:33.975916 (+ 347us) server_negotiation.cc:407] Connection header received
0927 15:02:33.975993 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:33.976000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:33.976208 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:33.976352 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:33.976564 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.977658 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.979104 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:33.979886 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:33.979984 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:33.980097 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:33.980933 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 15:02:33.981037 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":145,"threads_started":1}
I20260927 15:02:33.983155 4875 tablet_service.cc:1467] Processing CreateTablet for tablet a1f3552c2dec4440b4991ce0ba8ace1a (DEFAULT_TABLE table=test-table [id=b6eae33c0d2a48f8b8dbdd28c74eacec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:33.984094 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f3552c2dec4440b4991ce0ba8ace1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:33.984058 4742 tablet_service.cc:1467] Processing CreateTablet for tablet a1f3552c2dec4440b4991ce0ba8ace1a (DEFAULT_TABLE table=test-table [id=b6eae33c0d2a48f8b8dbdd28c74eacec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:33.984058 4609 tablet_service.cc:1467] Processing CreateTablet for tablet a1f3552c2dec4440b4991ce0ba8ace1a (DEFAULT_TABLE table=test-table [id=b6eae33c0d2a48f8b8dbdd28c74eacec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:33.985028 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f3552c2dec4440b4991ce0ba8ace1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:33.985028 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f3552c2dec4440b4991ce0ba8ace1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:33.989954 4958 tablet_bootstrap.cc:492] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: Bootstrap starting.
I20260927 15:02:33.991627 4959 tablet_bootstrap.cc:492] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: Bootstrap starting.
I20260927 15:02:33.992411 4960 tablet_bootstrap.cc:492] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: Bootstrap starting.
I20260927 15:02:33.993024 4958 tablet_bootstrap.cc:654] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:33.993691 4959 tablet_bootstrap.cc:654] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:33.994053 4960 tablet_bootstrap.cc:654] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:33.994362 4958 log.cc:824] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:33.994903 4959 log.cc:824] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:33.995177 4960 log.cc:824] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:33.996393 4958 tablet_bootstrap.cc:492] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: No bootstrap required, opened a new log
I20260927 15:02:33.996642 4958 ts_tablet_manager.cc:1397] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 15:02:33.996820 4959 tablet_bootstrap.cc:492] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: No bootstrap required, opened a new log
I20260927 15:02:33.997001 4959 ts_tablet_manager.cc:1397] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:02:33.998965 4960 tablet_bootstrap.cc:492] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: No bootstrap required, opened a new log
I20260927 15:02:33.999135 4960 ts_tablet_manager.cc:1397] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 15:02:33.999822 4958 raft_consensus.cc:348] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.000072 4958 raft_consensus.cc:374] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:34.000165 4958 raft_consensus.cc:729] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2366e9f71a4bf2b0ef53733971a801, State: Initialized, Role: FOLLOWER
I20260927 15:02:34.000625 4958 consensus_queue.cc:260] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.000694 4959 raft_consensus.cc:348] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.000960 4959 raft_consensus.cc:374] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:34.001019 4959 raft_consensus.cc:729] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bbf996825448e890270b05f37bf139, State: Initialized, Role: FOLLOWER
I20260927 15:02:34.001479 4941 heartbeater.cc:503] Master 127.0.98.62:39537 was elected leader, sending a full tablet report...
I20260927 15:02:34.001492 4959 consensus_queue.cc:260] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.001700 4958 ts_tablet_manager.cc:1428] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:02:34.002434 4959 ts_tablet_manager.cc:1428] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 15:02:34.002748 4675 heartbeater.cc:503] Master 127.0.98.62:39537 was elected leader, sending a full tablet report...
I20260927 15:02:34.003034 4960 raft_consensus.cc:348] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.003310 4960 raft_consensus.cc:374] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:34.003360 4960 raft_consensus.cc:729] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151c7f3b80f94e1994833f7b9426bd02, State: Initialized, Role: FOLLOWER
I20260927 15:02:34.003798 4960 consensus_queue.cc:260] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.004482 4808 heartbeater.cc:503] Master 127.0.98.62:39537 was elected leader, sending a full tablet report...
I20260927 15:02:34.004848 4960 ts_tablet_manager.cc:1428] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 15:02:34.007745 4676 tablet.cc:2369] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:34.174520 4942 tablet.cc:2369] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:34.217986 4809 tablet.cc:2369] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:34.263410 4966 raft_consensus.cc:482] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:34.263710 4966 raft_consensus.cc:504] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.265200 4966 leader_election.cc:288] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef2366e9f71a4bf2b0ef53733971a801 (127.0.98.3:42693), 44bbf996825448e890270b05f37bf139 (127.0.98.1:35261)
I20260927 15:02:34.269512 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.265546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59311 (local address 127.0.98.1:35261)
0927 15:02:34.265673 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:34.265678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:34.265876 (+ 198us) server_negotiation.cc:407] Connection header received
0927 15:02:34.266027 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:34.266030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:34.266088 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:34.266175 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:34.266462 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.267153 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.268463 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.268938 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.269018 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:34.269069 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:34.269299 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:02:34.269376 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:02:34.269510 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.265435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35261 (local address 127.0.98.2:59311)
0927 15:02:34.265741 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:34.265758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:34.265902 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:34.266195 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:34.266198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:34.266215 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:34.266347 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:34.266353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.267268 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.267272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:34.268329 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:34.268338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.269050 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.269054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:34.269152 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:34.269179 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:02:34.269257 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:02:34.269334 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":17,"thread_start_us":76,"threads_started":1}
I20260927 15:02:34.269732 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.265444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42693 (local address 127.0.98.2:49933)
0927 15:02:34.265890 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:34.265906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:34.266034 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:34.266637 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:34.266640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:34.266660 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:34.266778 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:34.266797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.267670 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.267673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:34.268561 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:34.268570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.269310 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.269313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:34.269409 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:34.269432 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:02:34.269507 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:02:34.269587 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":19,"thread_start_us":97,"threads_started":1}
I20260927 15:02:34.269749 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.265636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49933 (local address 127.0.98.3:42693)
0927 15:02:34.265868 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:34.265873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:34.266017 (+ 144us) server_negotiation.cc:407] Connection header received
0927 15:02:34.266490 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:34.266492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:34.266534 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:34.266591 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:34.266921 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.267554 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.268684 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.269175 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.269287 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:34.269344 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:34.269546 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 15:02:34.269624 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 15:02:34.270489 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f3552c2dec4440b4991ce0ba8ace1a" candidate_uuid: "151c7f3b80f94e1994833f7b9426bd02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bbf996825448e890270b05f37bf139" is_pre_election: true
I20260927 15:02:34.270520 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f3552c2dec4440b4991ce0ba8ace1a" candidate_uuid: "151c7f3b80f94e1994833f7b9426bd02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2366e9f71a4bf2b0ef53733971a801" is_pre_election: true
I20260927 15:02:34.270828 4629 raft_consensus.cc:2454] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151c7f3b80f94e1994833f7b9426bd02 in term 0.
I20260927 15:02:34.271030 4895 raft_consensus.cc:2454] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151c7f3b80f94e1994833f7b9426bd02 in term 0.
I20260927 15:02:34.271337 4695 leader_election.cc:302] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151c7f3b80f94e1994833f7b9426bd02, 44bbf996825448e890270b05f37bf139; no voters:
I20260927 15:02:34.271617 4966 raft_consensus.cc:2781] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:34.271700 4966 raft_consensus.cc:482] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:34.271742 4966 raft_consensus.cc:3037] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:34.294559 4966 raft_consensus.cc:504] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.295249 4966 leader_election.cc:288] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [CANDIDATE]: Term 1 election: Requested vote from peers ef2366e9f71a4bf2b0ef53733971a801 (127.0.98.3:42693), 44bbf996825448e890270b05f37bf139 (127.0.98.1:35261)
I20260927 15:02:34.295828 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f3552c2dec4440b4991ce0ba8ace1a" candidate_uuid: "151c7f3b80f94e1994833f7b9426bd02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bbf996825448e890270b05f37bf139"
I20260927 15:02:34.295830 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f3552c2dec4440b4991ce0ba8ace1a" candidate_uuid: "151c7f3b80f94e1994833f7b9426bd02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2366e9f71a4bf2b0ef53733971a801"
I20260927 15:02:34.296000 4895 raft_consensus.cc:3037] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:34.296005 4629 raft_consensus.cc:3037] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:34.297319 4629 raft_consensus.cc:2454] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151c7f3b80f94e1994833f7b9426bd02 in term 1.
I20260927 15:02:34.297706 4895 raft_consensus.cc:2454] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151c7f3b80f94e1994833f7b9426bd02 in term 1.
I20260927 15:02:34.297757 4695 leader_election.cc:302] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151c7f3b80f94e1994833f7b9426bd02, 44bbf996825448e890270b05f37bf139; no voters:
I20260927 15:02:34.297991 4966 raft_consensus.cc:2781] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:34.298254 4966 raft_consensus.cc:686] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 1 LEADER]: Becoming Leader. State: Replica: 151c7f3b80f94e1994833f7b9426bd02, State: Running, Role: LEADER
I20260927 15:02:34.298640 4966 consensus_queue.cc:237] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } }
I20260927 15:02:34.302028 4485 catalog_manager.cc:5495] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 151c7f3b80f94e1994833f7b9426bd02 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "151c7f3b80f94e1994833f7b9426bd02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "151c7f3b80f94e1994833f7b9426bd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42569 } health_report { overall_health: HEALTHY } } }
I20260927 15:02:34.343776 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.340110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.0.98.2:42569)
0927 15:02:34.340252 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:34.340257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:34.340300 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:02:34.340385 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:34.340387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:34.340441 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:34.340552 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:34.340837 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.341353 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.342318 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:34.342531 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:34.342639 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:34.342952 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:34.343407 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:34.343541 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:34.343581 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 15:02:34.343633 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:02:34.349551 4895 raft_consensus.cc:1261] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 1 FOLLOWER]: Refusing update from remote peer 151c7f3b80f94e1994833f7b9426bd02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:34.349552 4629 raft_consensus.cc:1261] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 1 FOLLOWER]: Refusing update from remote peer 151c7f3b80f94e1994833f7b9426bd02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:34.350333 4966 consensus_queue.cc:1035] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2366e9f71a4bf2b0ef53733971a801" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:34.350546 4969 consensus_queue.cc:1035] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44bbf996825448e890270b05f37bf139" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:34.358594 4974 mvcc.cc:205] Tried to move back new op lower bound from 7333975467408818176 to 7333975467210194944. Current Snapshot: MvccSnapshot[applied={T|T < 7333975467408818176}]
I20260927 15:02:34.360468 4973 mvcc.cc:205] Tried to move back new op lower bound from 7333975467408818176 to 7333975467210194944. Current Snapshot: MvccSnapshot[applied={T|T < 7333975467408818176}]
I20260927 15:02:34.366605 4485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:34.366825 4485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:34.369407 4485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15edc5715b7b4c898edb7c126c50455c: Sending DeleteTablet for 3 replicas of tablet a1f3552c2dec4440b4991ce0ba8ace1a
I20260927 15:02:34.370374 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f3552c2dec4440b4991ce0ba8ace1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33966
I20260927 15:02:34.370357 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f3552c2dec4440b4991ce0ba8ace1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36558
I20260927 15:02:34.370949 4981 tablet_replica.cc:330] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: stopping tablet replica
I20260927 15:02:34.371007 4982 tablet_replica.cc:330] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139: stopping tablet replica
I20260927 15:02:34.371064 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f3552c2dec4440b4991ce0ba8ace1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46852
I20260927 15:02:34.371485 4981 raft_consensus.cc:2229] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:34.371485 4982 raft_consensus.cc:2229] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:34.371841 4982 raft_consensus.cc:2258] T a1f3552c2dec4440b4991ce0ba8ace1a P 44bbf996825448e890270b05f37bf139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:34.372016 4981 raft_consensus.cc:2258] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:34.372252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4547
I20260927 15:02:34.374348 4981 ts_tablet_manager.cc:1905] T a1f3552c2dec4440b4991ce0ba8ace1a P 151c7f3b80f94e1994833f7b9426bd02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:02:34.378666 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35261 (error 108)
I20260927 15:02:34.378858 4983 tablet_replica.cc:330] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: stopping tablet replica
I20260927 15:02:34.379257 4983 raft_consensus.cc:2229] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:34.379478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4680
I20260927 15:02:34.380465 4983 raft_consensus.cc:2258] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:34.382313 4983 ts_tablet_manager.cc:1905] T a1f3552c2dec4440b4991ce0ba8ace1a P ef2366e9f71a4bf2b0ef53733971a801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:34.385380 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.385072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35261 (local address 127.0.0.1:33980)
0927 15:02:34.385199 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:34.385286 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 15:02:34.385599 4469 catalog_manager.cc:4584] TS 44bbf996825448e890270b05f37bf139 (127.0.98.1:35261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1f3552c2dec4440b4991ce0ba8ace1a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35261: connect: Connection refused (error 111)
I20260927 15:02:34.387203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4813
I20260927 15:02:34.388360 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.388070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42569 (local address 127.0.0.1:36568)
0927 15:02:34.388192 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:34.388251 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:02:34.432807 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.432459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42569 (local address 127.0.0.1:36578)
0927 15:02:34.432613 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:34.432686 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:02:34.433153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4454
2026-09-27T15:02:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2656 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T15:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:34Z Disabled control of system clock
I20260927 15:02:34.466305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46433
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:02:34.576424 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:34.576764 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:34.576809 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:34.581229 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:34.581293 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:34.581312 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:34.581332 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:34.581350 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:34.586514 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:34.587788 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:34.589170 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:34.598296 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:34.598675 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:34.599329 4988 server_base.cc:1034] running on GCE node
W20260927 15:02:34.599345 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:34.600031 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:34.601537 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:34.602762 4988 hybrid_clock.cc:648] HybridClock initialized: now 1790521354602737 us; error 42 us; skew 500 ppm
I20260927 15:02:34.604907 4988 webserver.cc:460] Webserver started at http://127.0.98.62:37183/ using document root <none> and password file <none>
I20260927 15:02:34.605505 4988 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:34.605571 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:34.605903 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:34.608011 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "2aa70e36079247ba87e2291541f0502c"
format_stamp: "Formatted at 2026-09-27 15:02:34 on dist-test-slave-ml7t"
server_key: "a4dad3f55a3e4abc863b419f205223d4"
server_key_iv: "91db51d3f29e9ee914de943464e57cd9"
server_key_version: "encryptionkey@0"
I20260927 15:02:34.608611 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "2aa70e36079247ba87e2291541f0502c"
format_stamp: "Formatted at 2026-09-27 15:02:34 on dist-test-slave-ml7t"
server_key: "a4dad3f55a3e4abc863b419f205223d4"
server_key_iv: "91db51d3f29e9ee914de943464e57cd9"
server_key_version: "encryptionkey@0"
I20260927 15:02:34.612458 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:02:34.615015 5002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:34.616207 4988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:34.616365 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "2aa70e36079247ba87e2291541f0502c"
format_stamp: "Formatted at 2026-09-27 15:02:34 on dist-test-slave-ml7t"
server_key: "a4dad3f55a3e4abc863b419f205223d4"
server_key_iv: "91db51d3f29e9ee914de943464e57cd9"
server_key_version: "encryptionkey@0"
I20260927 15:02:34.616490 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:34.621173 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:34.621884 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:34.622071 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:34.629047 4988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46433
I20260927 15:02:34.629055 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46433 every 8 connection(s)
I20260927 15:02:34.630026 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:34.631821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 4988
I20260927 15:02:34.631963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:34.632253 392 external_mini_cluster.cc:1428] Setting key 8ef0f9df70146096ac116bb50a7809fe
I20260927 15:02:34.633450 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:34.638551 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.633608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.98.62:46433)
0927 15:02:34.634418 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:34.634437 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:34.634480 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:02:34.635388 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:34.635411 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:34.635869 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:34.636279 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:34.637021 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:34.637026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:34.637040 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:34.637053 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:34.637118 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:34.637138 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:34.637295 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:34.637365 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:02:34.637612 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":199,"threads_started":1}
I20260927 15:02:34.640468 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c: Bootstrap starting.
I20260927 15:02:34.642895 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:34.643625 5055 log.cc:824] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:34.645546 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c: No bootstrap required, opened a new log
I20260927 15:02:34.649021 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa70e36079247ba87e2291541f0502c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46433 } }
I20260927 15:02:34.649322 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:34.649395 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa70e36079247ba87e2291541f0502c, State: Initialized, Role: FOLLOWER
I20260927 15:02:34.649917 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa70e36079247ba87e2291541f0502c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46433 } }
I20260927 15:02:34.650079 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:34.650146 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:34.650245 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:34.651176 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa70e36079247ba87e2291541f0502c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46433 } }
I20260927 15:02:34.651522 5055 leader_election.cc:302] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aa70e36079247ba87e2291541f0502c; no voters:
I20260927 15:02:34.651841 5055 leader_election.cc:288] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:34.651989 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:34.652243 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [term 1 LEADER]: Becoming Leader. State: Replica: 2aa70e36079247ba87e2291541f0502c, State: Running, Role: LEADER
I20260927 15:02:34.652577 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa70e36079247ba87e2291541f0502c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46433 } }
I20260927 15:02:34.653116 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:34.654318 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aa70e36079247ba87e2291541f0502c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa70e36079247ba87e2291541f0502c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46433 } } }
I20260927 15:02:34.654353 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aa70e36079247ba87e2291541f0502c. Latest consensus state: current_term: 1 leader_uuid: "2aa70e36079247ba87e2291541f0502c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa70e36079247ba87e2291541f0502c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46433 } } }
I20260927 15:02:34.654448 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:34.654451 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:34.654793 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:34.657622 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:34.663081 5069 catalog_manager.cc:1347] Generated new cluster ID: 25de13c2ec494e71a52dc09a9580d29e
I20260927 15:02:34.663160 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:34.681509 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:34.681897 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:34.690308 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c: Generated new TSK 0
I20260927 15:02:34.691058 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:02:34.702502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46433
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:02:34.812565 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:34.812820 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:34.812860 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:34.816867 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:34.816917 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:34.817004 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:34.821898 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:34.822980 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:34.824411 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:34.834314 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:34.834329 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:34.834347 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:34.834384 5079 server_base.cc:1034] running on GCE node
I20260927 15:02:34.834916 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:34.835608 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:34.836773 5079 hybrid_clock.cc:648] HybridClock initialized: now 1790521354836752 us; error 32 us; skew 500 ppm
I20260927 15:02:34.838857 5079 webserver.cc:460] Webserver started at http://127.0.98.1:43441/ using document root <none> and password file <none>
I20260927 15:02:34.839450 5079 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:34.839512 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:34.839753 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:34.841902 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "bac7da5b21f343bd888e443890a90b96"
format_stamp: "Formatted at 2026-09-27 15:02:34 on dist-test-slave-ml7t"
server_key: "6154bc5a93cff2164ffb2da287a8f963"
server_key_iv: "f6d4d36527eb8ab0378066a6b3a7b5dd"
server_key_version: "encryptionkey@0"
I20260927 15:02:34.842466 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bac7da5b21f343bd888e443890a90b96"
format_stamp: "Formatted at 2026-09-27 15:02:34 on dist-test-slave-ml7t"
server_key: "6154bc5a93cff2164ffb2da287a8f963"
server_key_iv: "f6d4d36527eb8ab0378066a6b3a7b5dd"
server_key_version: "encryptionkey@0"
I20260927 15:02:34.846195 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:34.848596 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:34.849823 5079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:34.849951 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "bac7da5b21f343bd888e443890a90b96"
format_stamp: "Formatted at 2026-09-27 15:02:34 on dist-test-slave-ml7t"
server_key: "6154bc5a93cff2164ffb2da287a8f963"
server_key_iv: "f6d4d36527eb8ab0378066a6b3a7b5dd"
server_key_version: "encryptionkey@0"
I20260927 15:02:34.850061 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:34.854919 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:34.855545 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:34.855736 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:34.856302 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:34.857209 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:34.857255 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:34.857293 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:34.857350 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:34.865751 5079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36547
I20260927 15:02:34.865796 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36547 every 8 connection(s)
I20260927 15:02:34.866664 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:34.868311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5079
I20260927 15:02:34.868434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:34.868716 392 external_mini_cluster.cc:1428] Setting key 4b7e9670b9e5d83c65d10788ad82d349
I20260927 15:02:34.870776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46433
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:02:34.873584 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.868481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51243 (local address 127.0.98.62:46433)
0927 15:02:34.868624 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:34.868628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:34.870164 (+ 1536us) server_negotiation.cc:407] Connection header received
0927 15:02:34.871719 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:34.871722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:34.871759 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:34.871838 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:34.873023 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:34.873027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:34.873029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:34.873043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:34.873066 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:34.873073 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:34.873186 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:34.873348 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:02:34.873397 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:02:34.874589 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:34.869012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46433 (local address 127.0.98.1:51243)
0927 15:02:34.870017 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:34.870073 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:34.871508 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:34.872002 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:34.872009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:34.872135 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:34.872766 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:34.872768 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:34.872812 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:34.872879 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:34.873172 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:34.873180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:34.873185 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:02:34.873359 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 15:02:34.873597 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":153,"threads_started":1}
I20260927 15:02:34.875744 5207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46433
I20260927 15:02:34.876009 5207 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:34.876504 5207 heartbeater.cc:511] Master 127.0.98.62:46433 requested a full tablet report, sending...
I20260927 15:02:34.878226 5019 ts_manager.cc:194] Registered new tserver with Master: bac7da5b21f343bd888e443890a90b96 (127.0.98.1:36547)
W20260927 15:02:34.981679 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:34.982000 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:34.982085 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:34.986230 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:34.986302 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:34.986398 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:34.991068 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:34.992089 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:34.993409 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:35.002727 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:35.002722 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:35.002753 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:35.003136 5210 server_base.cc:1034] running on GCE node
I20260927 15:02:35.003500 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:35.004138 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:35.005348 5210 hybrid_clock.cc:648] HybridClock initialized: now 1790521355005323 us; error 30 us; skew 500 ppm
I20260927 15:02:35.007325 5210 webserver.cc:460] Webserver started at http://127.0.98.2:33763/ using document root <none> and password file <none>
I20260927 15:02:35.007920 5210 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:35.008014 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:35.008247 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:35.010366 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b8eea7152d247dfb5f6a39ef27e40b7"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "45a496c9fcbb7e5194ff896910448ea1"
server_key_iv: "e8fd16159011b5b7ed69860539dc9b72"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.010941 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b8eea7152d247dfb5f6a39ef27e40b7"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "45a496c9fcbb7e5194ff896910448ea1"
server_key_iv: "e8fd16159011b5b7ed69860539dc9b72"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.014560 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:02:35.016971 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.018129 5210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:35.018257 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "9b8eea7152d247dfb5f6a39ef27e40b7"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "45a496c9fcbb7e5194ff896910448ea1"
server_key_iv: "e8fd16159011b5b7ed69860539dc9b72"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.018366 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:35.023135 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:35.023705 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:35.023881 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:35.024411 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:35.025257 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:35.025346 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.025417 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:35.025455 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.033756 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35689
I20260927 15:02:35.033854 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35689 every 8 connection(s)
I20260927 15:02:35.034678 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:35.037031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5210
I20260927 15:02:35.037175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:35.037465 392 external_mini_cluster.cc:1428] Setting key 6f8ebce3d691547bbed5a3433a6ea48b
I20260927 15:02:35.039451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46433
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:02:35.041112 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.036438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59649 (local address 127.0.98.62:46433)
0927 15:02:35.036595 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.036600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.037861 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 15:02:35.039313 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.039316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.039354 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.039454 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.040629 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.040633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.040635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.040653 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.040672 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.040680 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.040801 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.040937 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.040978 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:02:35.042044 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.036837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46433 (local address 127.0.98.2:59649)
0927 15:02:35.037662 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.037705 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.039098 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.039654 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.039665 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.039766 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:35.040379 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:35.040380 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:35.040420 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:35.040485 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:35.040788 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:35.040796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:35.040802 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:02:35.040980 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 15:02:35.041301 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":145,"threads_started":1}
I20260927 15:02:35.043113 5338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46433
I20260927 15:02:35.043344 5338 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:35.043726 5338 heartbeater.cc:511] Master 127.0.98.62:46433 requested a full tablet report, sending...
I20260927 15:02:35.044760 5019 ts_manager.cc:194] Registered new tserver with Master: 9b8eea7152d247dfb5f6a39ef27e40b7 (127.0.98.2:35689)
W20260927 15:02:35.152268 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:35.152516 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:35.152554 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:35.156837 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:35.156893 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:35.156965 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:35.161661 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:35.162712 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:35.163996 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:35.173630 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:35.173866 5341 server_base.cc:1034] running on GCE node
W20260927 15:02:35.173630 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:35.173691 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:35.174391 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:35.174959 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:35.176142 5341 hybrid_clock.cc:648] HybridClock initialized: now 1790521355176127 us; error 27 us; skew 500 ppm
I20260927 15:02:35.178004 5341 webserver.cc:460] Webserver started at http://127.0.98.3:38313/ using document root <none> and password file <none>
I20260927 15:02:35.178570 5341 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:35.178660 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:35.178896 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:35.180980 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "f4f28c0d5eb54356a8fe9811dde97069"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "a5f84fd490ff58de50d64e7fc6e297f3"
server_key_iv: "9c22f7ec1fd2a88f403eece59355f47c"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.181565 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f4f28c0d5eb54356a8fe9811dde97069"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "a5f84fd490ff58de50d64e7fc6e297f3"
server_key_iv: "9c22f7ec1fd2a88f403eece59355f47c"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.185318 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:02:35.187909 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.189100 5341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:02:35.189239 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "f4f28c0d5eb54356a8fe9811dde97069"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "a5f84fd490ff58de50d64e7fc6e297f3"
server_key_iv: "9c22f7ec1fd2a88f403eece59355f47c"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.189356 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:35.194007 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:35.194593 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:35.194777 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:35.195293 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:35.196163 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:35.196233 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.196300 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:35.196349 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.204640 5341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42485
I20260927 15:02:35.204661 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42485 every 8 connection(s)
I20260927 15:02:35.205500 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:35.206349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5341
I20260927 15:02:35.206470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:35.206745 392 external_mini_cluster.cc:1428] Setting key 8fd265febad572f47afc6455ecc8bdd9
I20260927 15:02:35.211745 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.207372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50787 (local address 127.0.98.62:46433)
0927 15:02:35.207496 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.207499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.208831 (+ 1332us) server_negotiation.cc:407] Connection header received
0927 15:02:35.210163 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.210166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.210206 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.210283 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.211273 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.211279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.211282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.211293 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.211313 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.211322 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.211428 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.211575 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.211617 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:02:35.212567 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.207953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46433 (local address 127.0.98.3:50787)
0927 15:02:35.208680 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.208728 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.209967 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.210439 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.210446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.210529 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:35.211034 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:35.211035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:35.211072 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:35.211136 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:35.211415 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:35.211422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:35.211428 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:02:35.211603 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:02:35.211830 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20260927 15:02:35.213610 5469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46433
I20260927 15:02:35.213848 5469 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:35.214231 5469 heartbeater.cc:511] Master 127.0.98.62:46433 requested a full tablet report, sending...
I20260927 15:02:35.215204 5019 ts_manager.cc:194] Registered new tserver with Master: f4f28c0d5eb54356a8fe9811dde97069 (127.0.98.3:42485)
I20260927 15:02:35.220486 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:02:35.223945 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.222363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.0.98.62:46433)
0927 15:02:35.222519 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.222523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.222689 (+ 166us) server_negotiation.cc:407] Connection header received
0927 15:02:35.222795 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.222797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.222829 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.222937 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.223603 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.223605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.223607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.223614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.223629 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.223635 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.223700 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.223765 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.223794 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:02:35.226696 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:35.229899 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:35.241034 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.237928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42485 (local address 127.0.0.1:43206)
0927 15:02:35.238875 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.238927 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.239027 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.239654 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.239660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.239733 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:35.240289 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:35.240291 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:35.240315 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:35.240368 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:35.240668 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:35.240676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:35.240682 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:02:35.240800 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:02:35.240869 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":80,"threads_started":1}
I20260927 15:02:35.241596 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.238026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.0.98.3:42485)
0927 15:02:35.239148 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.239159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.239182 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:02:35.239237 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.239245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.239482 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.239660 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.240490 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.240496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.240508 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.240519 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.240563 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.240582 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.240695 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.241358 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.241429 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":81,"threads_started":1}
I20260927 15:02:35.241680 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.238383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.0.98.1:36547)
0927 15:02:35.238652 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.238664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.239035 (+ 371us) server_negotiation.cc:407] Connection header received
0927 15:02:35.239219 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.239227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.239448 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.239579 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.241013 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.241022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.241037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.241051 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.241118 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.241152 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.241247 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.241417 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.241490 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260927 15:02:35.242090 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.238000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36547 (local address 127.0.0.1:59334)
0927 15:02:35.238875 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.238927 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.239086 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.240316 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.240318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.240332 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:35.240811 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:35.240815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:35.240827 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:35.240857 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:35.241315 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:35.241319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:35.241322 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:02:35.241935 (+ 613us) client_negotiation.cc:240] Negotiation successful
0927 15:02:35.241970 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":95,"threads_started":1}
I20260927 15:02:35.242223 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.238597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35689 (local address 127.0.0.1:59524)
0927 15:02:35.239346 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.239359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.239436 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.240277 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.240280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.240297 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:35.240916 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:35.240920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:35.240928 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:35.240953 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:35.242002 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:35.242005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:35.242007 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 15:02:35.242084 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:02:35.242115 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":99,"threads_started":1}
I20260927 15:02:35.242590 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.239013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.0.98.2:35689)
0927 15:02:35.239291 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.239302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.239444 (+ 142us) server_negotiation.cc:407] Connection header received
0927 15:02:35.239532 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.239537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.239686 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.239814 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.241748 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.241753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.241762 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.241799 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.241838 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.241864 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.241960 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.242110 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.242435 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260927 15:02:35.244655 5403 tablet_service.cc:1467] Processing CreateTablet for tablet 41ed76998e5448fb8c174cb2df6713e7 (DEFAULT_TABLE table=test-table [id=0de7d875515749b69bfb0612319ed7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:35.244709 5141 tablet_service.cc:1467] Processing CreateTablet for tablet 41ed76998e5448fb8c174cb2df6713e7 (DEFAULT_TABLE table=test-table [id=0de7d875515749b69bfb0612319ed7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:35.244839 5272 tablet_service.cc:1467] Processing CreateTablet for tablet 41ed76998e5448fb8c174cb2df6713e7 (DEFAULT_TABLE table=test-table [id=0de7d875515749b69bfb0612319ed7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:35.245822 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ed76998e5448fb8c174cb2df6713e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:35.245824 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ed76998e5448fb8c174cb2df6713e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:35.245827 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ed76998e5448fb8c174cb2df6713e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:35.251832 5486 tablet_bootstrap.cc:492] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Bootstrap starting.
I20260927 15:02:35.253495 5488 tablet_bootstrap.cc:492] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96: Bootstrap starting.
I20260927 15:02:35.253528 5487 tablet_bootstrap.cc:492] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Bootstrap starting.
I20260927 15:02:35.254522 5486 tablet_bootstrap.cc:654] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:35.255282 5486 log.cc:824] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:35.255700 5487 tablet_bootstrap.cc:654] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:35.255700 5488 tablet_bootstrap.cc:654] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:35.256515 5487 log.cc:824] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:35.256515 5488 log.cc:824] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:35.257191 5486 tablet_bootstrap.cc:492] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: No bootstrap required, opened a new log
I20260927 15:02:35.257391 5486 ts_tablet_manager.cc:1397] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:02:35.258391 5487 tablet_bootstrap.cc:492] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: No bootstrap required, opened a new log
I20260927 15:02:35.258391 5488 tablet_bootstrap.cc:492] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96: No bootstrap required, opened a new log
I20260927 15:02:35.258569 5487 ts_tablet_manager.cc:1397] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 15:02:35.258569 5488 ts_tablet_manager.cc:1397] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 15:02:35.261351 5486 raft_consensus.cc:348] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.261591 5486 raft_consensus.cc:374] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:35.261642 5486 raft_consensus.cc:729] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8eea7152d247dfb5f6a39ef27e40b7, State: Initialized, Role: FOLLOWER
I20260927 15:02:35.262133 5486 consensus_queue.cc:260] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.263010 5486 ts_tablet_manager.cc:1428] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:02:35.262948 5487 raft_consensus.cc:348] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.262946 5488 raft_consensus.cc:348] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.263187 5487 raft_consensus.cc:374] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:35.263187 5488 raft_consensus.cc:374] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:35.263247 5488 raft_consensus.cc:729] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac7da5b21f343bd888e443890a90b96, State: Initialized, Role: FOLLOWER
I20260927 15:02:35.263273 5487 raft_consensus.cc:729] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f28c0d5eb54356a8fe9811dde97069, State: Initialized, Role: FOLLOWER
I20260927 15:02:35.263389 5338 heartbeater.cc:503] Master 127.0.98.62:46433 was elected leader, sending a full tablet report...
I20260927 15:02:35.263752 5487 consensus_queue.cc:260] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.263759 5488 consensus_queue.cc:260] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.264645 5469 heartbeater.cc:503] Master 127.0.98.62:46433 was elected leader, sending a full tablet report...
I20260927 15:02:35.264742 5487 ts_tablet_manager.cc:1428] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 15:02:35.264742 5488 ts_tablet_manager.cc:1428] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 15:02:35.265856 5207 heartbeater.cc:503] Master 127.0.98.62:46433 was elected leader, sending a full tablet report...
W20260927 15:02:35.285650 5339 tablet.cc:2369] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:35.367831 5208 tablet.cc:2369] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:35.374804 5492 raft_consensus.cc:482] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:35.375025 5492 raft_consensus.cc:504] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.376174 5492 leader_election.cc:288] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f28c0d5eb54356a8fe9811dde97069 (127.0.98.3:42485), bac7da5b21f343bd888e443890a90b96 (127.0.98.1:36547)
I20260927 15:02:35.378284 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.376574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34147 (local address 127.0.98.1:36547)
0927 15:02:35.376726 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.376731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.376996 (+ 265us) server_negotiation.cc:407] Connection header received
0927 15:02:35.377044 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.377050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.377089 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.377166 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.377888 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.377892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.377895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.377907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.377928 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.377940 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.378024 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.378121 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.378156 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:02:35.378285 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.376566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35711 (local address 127.0.98.3:42485)
0927 15:02:35.376690 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.376696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.376778 (+ 82us) server_negotiation.cc:407] Connection header received
0927 15:02:35.376937 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.376940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.376998 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.377089 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.377844 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.377847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.377850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.377864 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.377886 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.377896 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.377994 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.378110 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.378153 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:02:35.378293 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.376498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42485 (local address 127.0.98.2:35711)
0927 15:02:35.376688 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.376704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.376829 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.377119 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.377121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.377150 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:35.377610 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:35.377611 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:35.377641 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:35.377676 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:35.377999 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:35.378002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:35.378007 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:02:35.378092 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:02:35.378134 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 15:02:35.378543 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.376500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36547 (local address 127.0.98.2:34147)
0927 15:02:35.376894 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.376909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.376988 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.377187 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.377193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.377208 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:35.377708 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:02:35.377712 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:35.377722 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:02:35.377747 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:35.378023 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:35.378027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:35.378032 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:02:35.378107 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:02:35.378136 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":49,"thread_start_us":134,"threads_started":1}
I20260927 15:02:35.378986 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ed76998e5448fb8c174cb2df6713e7" candidate_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f28c0d5eb54356a8fe9811dde97069" is_pre_election: true
I20260927 15:02:35.379191 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ed76998e5448fb8c174cb2df6713e7" candidate_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac7da5b21f343bd888e443890a90b96" is_pre_election: true
I20260927 15:02:35.379271 5423 raft_consensus.cc:2454] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8eea7152d247dfb5f6a39ef27e40b7 in term 0.
I20260927 15:02:35.379465 5161 raft_consensus.cc:2454] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8eea7152d247dfb5f6a39ef27e40b7 in term 0.
I20260927 15:02:35.379767 5228 leader_election.cc:302] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8eea7152d247dfb5f6a39ef27e40b7, f4f28c0d5eb54356a8fe9811dde97069; no voters:
I20260927 15:02:35.380043 5492 raft_consensus.cc:2781] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:35.380136 5492 raft_consensus.cc:482] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:35.380193 5492 raft_consensus.cc:3037] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:35.381179 5492 raft_consensus.cc:504] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.381512 5492 leader_election.cc:288] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [CANDIDATE]: Term 1 election: Requested vote from peers f4f28c0d5eb54356a8fe9811dde97069 (127.0.98.3:42485), bac7da5b21f343bd888e443890a90b96 (127.0.98.1:36547)
I20260927 15:02:35.381872 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ed76998e5448fb8c174cb2df6713e7" candidate_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac7da5b21f343bd888e443890a90b96"
I20260927 15:02:35.381872 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ed76998e5448fb8c174cb2df6713e7" candidate_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f28c0d5eb54356a8fe9811dde97069"
I20260927 15:02:35.381989 5423 raft_consensus.cc:3037] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:35.381990 5161 raft_consensus.cc:3037] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:35.383215 5423 raft_consensus.cc:2454] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8eea7152d247dfb5f6a39ef27e40b7 in term 1.
I20260927 15:02:35.383209 5161 raft_consensus.cc:2454] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8eea7152d247dfb5f6a39ef27e40b7 in term 1.
I20260927 15:02:35.383467 5228 leader_election.cc:302] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8eea7152d247dfb5f6a39ef27e40b7, f4f28c0d5eb54356a8fe9811dde97069; no voters:
I20260927 15:02:35.383685 5492 raft_consensus.cc:2781] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:35.383860 5492 raft_consensus.cc:686] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 1 LEADER]: Becoming Leader. State: Replica: 9b8eea7152d247dfb5f6a39ef27e40b7, State: Running, Role: LEADER
I20260927 15:02:35.384168 5492 consensus_queue.cc:237] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } }
I20260927 15:02:35.387280 5017 catalog_manager.cc:5495] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b8eea7152d247dfb5f6a39ef27e40b7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b8eea7152d247dfb5f6a39ef27e40b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35689 } health_report { overall_health: HEALTHY } } }
I20260927 15:02:35.431257 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.429730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.0.98.2:35689)
0927 15:02:35.429896 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.429899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.429912 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:02:35.430002 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.430005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.430061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.430131 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.430792 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.430796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.430798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.430809 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.430828 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.430836 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.430919 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.431058 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.431106 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:02:35.437148 5423 raft_consensus.cc:1261] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 1 FOLLOWER]: Refusing update from remote peer 9b8eea7152d247dfb5f6a39ef27e40b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:35.437150 5161 raft_consensus.cc:1261] T 41ed76998e5448fb8c174cb2df6713e7 P bac7da5b21f343bd888e443890a90b96 [term 1 FOLLOWER]: Refusing update from remote peer 9b8eea7152d247dfb5f6a39ef27e40b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:35.437896 5492 consensus_queue.cc:1035] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac7da5b21f343bd888e443890a90b96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:35.438072 5496 consensus_queue.cc:1035] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f28c0d5eb54356a8fe9811dde97069" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:35.449988 5500 mvcc.cc:205] Tried to move back new op lower bound from 7333975471863197696 to 7333975471656280064. Current Snapshot: MvccSnapshot[applied={T|T < 7333975471863197696}]
I20260927 15:02:35.452654 5017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:35.452829 5017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:35.455204 5017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2aa70e36079247ba87e2291541f0502c: Sending DeleteTablet for 3 replicas of tablet 41ed76998e5448fb8c174cb2df6713e7
I20260927 15:02:35.455883 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ed76998e5448fb8c174cb2df6713e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:35 UTC) from {username='slave'} at 127.0.0.1:59334
I20260927 15:02:35.455906 5403 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ed76998e5448fb8c174cb2df6713e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:35 UTC) from {username='slave'} at 127.0.0.1:43206
I20260927 15:02:35.456606 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ed76998e5448fb8c174cb2df6713e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:35 UTC) from {username='slave'} at 127.0.0.1:59524
I20260927 15:02:35.457068 5511 tablet_replica.cc:330] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: stopping tablet replica
I20260927 15:02:35.457109 5510 tablet_replica.cc:330] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: stopping tablet replica
I20260927 15:02:35.457276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5079
I20260927 15:02:35.457382 5510 raft_consensus.cc:2229] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:35.457388 5511 raft_consensus.cc:2229] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:35.457669 5510 raft_consensus.cc:2258] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:35.457716 5511 raft_consensus.cc:2258] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:35.459321 5510 ts_tablet_manager.cc:1905] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:35.459327 5511 ts_tablet_manager.cc:1905] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:35.462275 5510 ts_tablet_manager.cc:1918] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:35.462383 5510 log.cc:1196] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/41ed76998e5448fb8c174cb2df6713e7
I20260927 15:02:35.462644 5511 ts_tablet_manager.cc:1918] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:35.462702 5510 ts_tablet_manager.cc:1939] T 41ed76998e5448fb8c174cb2df6713e7 P f4f28c0d5eb54356a8fe9811dde97069: Deleting consensus metadata
I20260927 15:02:35.462749 5511 log.cc:1196] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/41ed76998e5448fb8c174cb2df6713e7
I20260927 15:02:35.463085 5511 ts_tablet_manager.cc:1939] T 41ed76998e5448fb8c174cb2df6713e7 P 9b8eea7152d247dfb5f6a39ef27e40b7: Deleting consensus metadata
I20260927 15:02:35.464131 5006 catalog_manager.cc:4841] TS f4f28c0d5eb54356a8fe9811dde97069 (127.0.98.3:42485): tablet 41ed76998e5448fb8c174cb2df6713e7 (table test-table [id=0de7d875515749b69bfb0612319ed7a1]) successfully deleted
I20260927 15:02:35.464600 5006 catalog_manager.cc:4841] TS 9b8eea7152d247dfb5f6a39ef27e40b7 (127.0.98.2:35689): tablet 41ed76998e5448fb8c174cb2df6713e7 (table test-table [id=0de7d875515749b69bfb0612319ed7a1]) successfully deleted
W20260927 15:02:35.467319 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36547 (error 108)
I20260927 15:02:35.468247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5210
I20260927 15:02:35.468473 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.468128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36547 (local address 127.0.0.1:59340)
0927 15:02:35.468282 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.468369 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:02:35.469059 5005 catalog_manager.cc:4584] TS bac7da5b21f343bd888e443890a90b96 (127.0.98.1:36547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41ed76998e5448fb8c174cb2df6713e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36547: connect: Connection refused (error 111)
I20260927 15:02:35.474488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5341
I20260927 15:02:35.480265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 4988
2026-09-27T15:02:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1043 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 15:02:35.501679 392 mini_oidc.cc:152] Starting JWKS server
I20260927 15:02:35.503295 392 webserver.cc:460] Webserver started at https://127.0.0.1:39581/ using document root <none> and password file <none>
I20260927 15:02:35.503494 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:02:35.503768 392 webserver.cc:460] Webserver started at http://127.0.0.1:46463/ using document root <none> and password file <none>
2026-09-27T15:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:35Z Disabled control of system clock
I20260927 15:02:35.512507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38951
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38951
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/testchainca.cert with env {}
W20260927 15:02:35.628296 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:35.628566 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:35.628629 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:35.632592 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:35.632661 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:35.632680 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:35.632700 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:35.632719 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:02:35.632738 5518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:02:35.632755 5518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39581/jwks.json
I20260927 15:02:35.638149 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:35.639163 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:35.640116 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:35.648830 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:35.648840 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:35.648922 5518 server_base.cc:1034] running on GCE node
W20260927 15:02:35.648840 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:35.649585 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:35.650784 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:35.651991 5518 hybrid_clock.cc:648] HybridClock initialized: now 1790521355651981 us; error 33 us; skew 500 ppm
I20260927 15:02:35.653973 5518 webserver.cc:460] Webserver started at http://127.0.98.62:40991/ using document root <none> and password file <none>
I20260927 15:02:35.654569 5518 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:35.654663 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:35.654899 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:35.656908 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "647269e150ca425c97549427a03a8349"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "58c87712f26c4e26f8ea8612104a769d"
server_key_iv: "f5ed0e4940862be1e277a914e98c725c"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.657481 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "647269e150ca425c97549427a03a8349"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "58c87712f26c4e26f8ea8612104a769d"
server_key_iv: "f5ed0e4940862be1e277a914e98c725c"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.661099 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:02:35.663472 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.664642 5518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 15:02:35.664781 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "647269e150ca425c97549427a03a8349"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "58c87712f26c4e26f8ea8612104a769d"
server_key_iv: "f5ed0e4940862be1e277a914e98c725c"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.664891 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:35.682751 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:35.683429 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:35.683630 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:35.690306 5518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38951
I20260927 15:02:35.690359 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38951 every 8 connection(s)
I20260927 15:02:35.691325 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:35.694859 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:35.698650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5518
I20260927 15:02:35.698786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:35.699106 392 external_mini_cluster.cc:1428] Setting key 72e25d38d846640cd2c0ac383a605cb7
I20260927 15:02:35.702164 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349: Bootstrap starting.
I20260927 15:02:35.704622 5585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:35.705395 5585 log.cc:824] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:35.707561 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349: No bootstrap required, opened a new log
I20260927 15:02:35.708485 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.700476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.0.98.62:38951)
0927 15:02:35.701127 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.701145 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.701181 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:02:35.701862 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.701881 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.702274 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.702646 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:35.703061 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:35.704083 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:35.704998 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:35.705473 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:35.705649 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:35.706575 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:35.706580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:35.706659 (+ 79us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:35.707068 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:35.707133 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:35.707155 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:35.707242 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:35.707408 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 15:02:35.707787 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260927 15:02:35.711030 5585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647269e150ca425c97549427a03a8349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260927 15:02:35.711263 5585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:35.711318 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:02:35.711325 5585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647269e150ca425c97549427a03a8349, State: Initialized, Role: FOLLOWER
I20260927 15:02:35.711763 5585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647269e150ca425c97549427a03a8349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260927 15:02:35.711882 5585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:35.711966 5585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:35.712087 5585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:35.713094 5585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647269e150ca425c97549427a03a8349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260927 15:02:35.713418 5585 leader_election.cc:302] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 647269e150ca425c97549427a03a8349; no voters:
I20260927 15:02:35.713693 5585 leader_election.cc:288] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:35.713829 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:35.714083 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [term 1 LEADER]: Becoming Leader. State: Replica: 647269e150ca425c97549427a03a8349, State: Running, Role: LEADER
I20260927 15:02:35.714403 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647269e150ca425c97549427a03a8349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260927 15:02:35.714854 5585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:35.716070 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "647269e150ca425c97549427a03a8349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647269e150ca425c97549427a03a8349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } } }
I20260927 15:02:35.716264 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:35.716146 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 647269e150ca425c97549427a03a8349. Latest consensus state: current_term: 1 leader_uuid: "647269e150ca425c97549427a03a8349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647269e150ca425c97549427a03a8349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } } }
I20260927 15:02:35.716625 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:35.717334 5602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:35.719910 5602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:35.725368 5602 catalog_manager.cc:1347] Generated new cluster ID: fa024cd6603240a3830a6f9b278c555c
I20260927 15:02:35.725433 5602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:35.739303 5602 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:35.740223 5602 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:35.744451 5602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 647269e150ca425c97549427a03a8349: Generated new TSK 0
I20260927 15:02:35.745198 5602 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:02:35.778640 5615 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 15:02:35.778775 5615 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:02:35.779428 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:35.761938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.0.98.62:38951)
0927 15:02:35.762084 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:35.762089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:35.762255 (+ 166us) server_negotiation.cc:407] Connection header received
0927 15:02:35.762453 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:35.762457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:35.762515 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:35.762632 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:02:35.762893 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:35.763732 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:35.764560 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:35.764914 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:35.765020 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:35.765354 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:02:35.778975 (+ 13621us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:02:35.779275 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:02:35.779493 5588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 15:02:35.780556 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:35.761833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38951 (local address 127.0.0.1:56772)
0927 15:02:35.762169 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:35.762182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:35.762294 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:35.762649 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:35.762652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:35.762658 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:02:35.762784 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:35.762789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:35.763834 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:35.763836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:35.764450 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:35.764458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:35.765021 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:35.765039 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:35.765107 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:35.765152 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:02:35.779209 (+ 14057us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:02:35.779414 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38951: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:38951: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 15:02:35.781333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5518
2026-09-27T15:02:35Z chronyd exiting
I20260927 15:02:35.818120 392 test_util.cc:175] -----------------------------------------------
I20260927 15:02:35.818228 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521328452215-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 15:02:35.819303 392 mini_oidc.cc:152] Starting JWKS server
I20260927 15:02:35.820195 392 webserver.cc:460] Webserver started at https://127.0.0.1:36465/ using document root <none> and password file <none>
I20260927 15:02:35.820330 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:02:35.820591 392 webserver.cc:460] Webserver started at http://127.0.0.1:33339/ using document root <none> and password file <none>
2026-09-27T15:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:35Z Disabled control of system clock
I20260927 15:02:35.829310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43937
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43937
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36465/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/testchainca.cert with env {}
W20260927 15:02:35.936821 5621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:35.937090 5621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:35.937131 5621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:35.941174 5621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:35.941233 5621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:35.941255 5621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:35.941272 5621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:35.941287 5621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:02:35.941352 5621 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:02:35.941380 5621 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36465/jwks.json
I20260927 15:02:35.946718 5621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36465/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:35.947726 5621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:35.948705 5621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:35.956739 5626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:35.956756 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:35.956784 5629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:35.957075 5621 server_base.cc:1034] running on GCE node
I20260927 15:02:35.957600 5621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:35.958710 5621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:35.959872 5621 hybrid_clock.cc:648] HybridClock initialized: now 1790521355959845 us; error 38 us; skew 500 ppm
I20260927 15:02:35.966956 5621 webserver.cc:460] Webserver started at http://127.0.98.62:35951/ using document root <none> and password file <none>
I20260927 15:02:35.967588 5621 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:35.967655 5621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:35.967907 5621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:35.970234 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "5cb8ee9c15b5400e87e73fdf0c6667df"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "8f16b405f64aa0a3dea902b6d6dfff32"
server_key_iv: "699bef6c072d7f4599916ab50f7db91f"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.970849 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "5cb8ee9c15b5400e87e73fdf0c6667df"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "8f16b405f64aa0a3dea902b6d6dfff32"
server_key_iv: "699bef6c072d7f4599916ab50f7db91f"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.974879 5621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:02:35.977314 5635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:35.978680 5621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:35.978830 5621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "5cb8ee9c15b5400e87e73fdf0c6667df"
format_stamp: "Formatted at 2026-09-27 15:02:35 on dist-test-slave-ml7t"
server_key: "8f16b405f64aa0a3dea902b6d6dfff32"
server_key_iv: "699bef6c072d7f4599916ab50f7db91f"
server_key_version: "encryptionkey@0"
I20260927 15:02:35.978948 5621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:36.000192 5621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:36.000964 5621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:36.001184 5621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:36.008786 5621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43937
I20260927 15:02:36.008852 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43937 every 8 connection(s)
I20260927 15:02:36.009872 5621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:36.013098 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:36.014914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5621
I20260927 15:02:36.015043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:36.015336 392 external_mini_cluster.cc:1428] Setting key a53c9e2fdc608a89f483289cfcf5d518
I20260927 15:02:36.019639 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df: Bootstrap starting.
I20260927 15:02:36.021977 5688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:36.022822 5688 log.cc:824] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:36.024636 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:36.016732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.98.62:43937)
0927 15:02:36.017369 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:36.017384 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:36.017418 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:02:36.018136 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:36.018154 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:36.018532 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:36.018885 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:36.019290 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.020365 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.021234 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.021763 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.021975 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.022820 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:36.022825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:36.022837 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:36.023310 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:36.023378 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:36.023402 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:36.023507 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:36.023684 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:02:36.023948 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260927 15:02:36.024856 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df: No bootstrap required, opened a new log
I20260927 15:02:36.027323 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:02:36.028584 5688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260927 15:02:36.028846 5688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:36.028906 5688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb8ee9c15b5400e87e73fdf0c6667df, State: Initialized, Role: FOLLOWER
I20260927 15:02:36.029328 5688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260927 15:02:36.029445 5688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:36.029490 5688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:36.029558 5688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:36.030445 5688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260927 15:02:36.030732 5688 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb8ee9c15b5400e87e73fdf0c6667df; no voters:
I20260927 15:02:36.031004 5688 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:36.031091 5693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:36.031340 5693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [term 1 LEADER]: Becoming Leader. State: Replica: 5cb8ee9c15b5400e87e73fdf0c6667df, State: Running, Role: LEADER
I20260927 15:02:36.031630 5693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } }
I20260927 15:02:36.032284 5688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:36.033589 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb8ee9c15b5400e87e73fdf0c6667df. Latest consensus state: current_term: 1 leader_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } } }
I20260927 15:02:36.033593 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb8ee9c15b5400e87e73fdf0c6667df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43937 } } }
I20260927 15:02:36.033743 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:36.033747 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:36.034155 5702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:36.036729 5702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:36.042024 5702 catalog_manager.cc:1347] Generated new cluster ID: 43c231c1e1b14e1386eeabf690a9ab85
I20260927 15:02:36.042095 5702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:36.066769 5702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:36.068112 5702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:36.077787 5702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb8ee9c15b5400e87e73fdf0c6667df: Generated new TSK 0
I20260927 15:02:36.078615 5702 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:02:36.104415 5718 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 15:02:36.104517 5718 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:02:36.105211 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:36.089390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.0.98.62:43937)
0927 15:02:36.089517 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:36.089522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:36.089672 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:02:36.089829 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:36.089833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:36.089889 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:36.090003 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:02:36.090215 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.091050 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.091885 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.092230 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.092333 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.092714 (+ 381us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:02:36.104802 (+ 12088us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:02:36.105063 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 15:02:36.105278 5691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 15:02:36.105257 5717 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:36.089273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43937 (local address 127.0.0.1:35214)
0927 15:02:36.089585 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:36.089596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:36.089696 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:36.090006 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:36.090008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:36.090014 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:02:36.090110 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:36.090122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.091170 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.091172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:36.091782 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:36.091789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.092370 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.092388 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:36.092465 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.092491 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:02:36.104986 (+ 12495us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:02:36.105125 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260927 15:02:36.106002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5621
2026-09-27T15:02:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 15:02:36.144454 392 mini_oidc.cc:152] Starting JWKS server
I20260927 15:02:36.145366 392 webserver.cc:460] Webserver started at https://127.0.0.1:38819/ using document root <none> and password file <none>
I20260927 15:02:36.145449 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:02:36.145700 392 webserver.cc:460] Webserver started at http://127.0.0.1:45899/ using document root <none> and password file <none>
2026-09-27T15:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:36Z Disabled control of system clock
I20260927 15:02:36.154090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39675
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39675
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38819/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:02:36.260267 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:36.260519 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:36.260555 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:36.264448 5724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:36.264500 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:36.264515 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:36.264532 5724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:36.264547 5724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:02:36.264562 5724 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:02:36.264576 5724 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38819/jwks.json
I20260927 15:02:36.269268 5724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38819/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:36.270315 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:36.271634 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:36.280395 5729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:36.280419 5732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:36.280437 5730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:36.280642 5724 server_base.cc:1034] running on GCE node
I20260927 15:02:36.281279 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:36.282374 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:36.283529 5724 hybrid_clock.cc:648] HybridClock initialized: now 1790521356283514 us; error 28 us; skew 500 ppm
I20260927 15:02:36.285339 5724 webserver.cc:460] Webserver started at http://127.0.98.62:46261/ using document root <none> and password file <none>
I20260927 15:02:36.285995 5724 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:36.286057 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:36.286288 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:36.288321 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "2abb37bbea994428805c5d63246b62e8"
format_stamp: "Formatted at 2026-09-27 15:02:36 on dist-test-slave-ml7t"
server_key: "bee467fb21252926c6fbcca75e0ed857"
server_key_iv: "6ae40db4044b23abf20d8a0a6f6f31d9"
server_key_version: "encryptionkey@0"
I20260927 15:02:36.288936 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "2abb37bbea994428805c5d63246b62e8"
format_stamp: "Formatted at 2026-09-27 15:02:36 on dist-test-slave-ml7t"
server_key: "bee467fb21252926c6fbcca75e0ed857"
server_key_iv: "6ae40db4044b23abf20d8a0a6f6f31d9"
server_key_version: "encryptionkey@0"
I20260927 15:02:36.292797 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 15:02:36.295442 5738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:36.296751 5724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:36.296896 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "2abb37bbea994428805c5d63246b62e8"
format_stamp: "Formatted at 2026-09-27 15:02:36 on dist-test-slave-ml7t"
server_key: "bee467fb21252926c6fbcca75e0ed857"
server_key_iv: "6ae40db4044b23abf20d8a0a6f6f31d9"
server_key_version: "encryptionkey@0"
I20260927 15:02:36.297021 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:36.316594 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:36.317389 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:36.317595 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:36.324474 5724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39675
I20260927 15:02:36.324486 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39675 every 8 connection(s)
I20260927 15:02:36.325659 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:36.329479 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:36.329506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5724
I20260927 15:02:36.329654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:36.330016 392 external_mini_cluster.cc:1428] Setting key 94ce4dd10b0f030cecd1e68d7424f27d
I20260927 15:02:36.335070 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8: Bootstrap starting.
I20260927 15:02:36.337211 5791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:36.337980 5791 log.cc:824] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:36.339979 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8: No bootstrap required, opened a new log
I20260927 15:02:36.340221 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:36.331348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.0.98.62:39675)
0927 15:02:36.332077 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:36.332103 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:36.332154 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:02:36.332909 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:36.332927 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:36.333360 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:36.333729 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:36.334245 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.335343 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.336371 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.336897 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.337099 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.338159 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:36.338169 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:36.338190 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:36.338746 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:36.338828 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:36.338860 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:36.338992 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:36.339151 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 15:02:36.339425 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":163,"threads_started":1}
I20260927 15:02:36.343355 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:02:36.343990 5791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb37bbea994428805c5d63246b62e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39675 } }
I20260927 15:02:36.344234 5791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:36.344296 5791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abb37bbea994428805c5d63246b62e8, State: Initialized, Role: FOLLOWER
I20260927 15:02:36.344714 5791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb37bbea994428805c5d63246b62e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39675 } }
I20260927 15:02:36.344833 5791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:36.344928 5791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:36.345034 5791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:36.346056 5791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb37bbea994428805c5d63246b62e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39675 } }
I20260927 15:02:36.346340 5791 leader_election.cc:302] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2abb37bbea994428805c5d63246b62e8; no voters:
I20260927 15:02:36.346591 5791 leader_election.cc:288] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:36.346751 5796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:36.346992 5796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2abb37bbea994428805c5d63246b62e8, State: Running, Role: LEADER
I20260927 15:02:36.347280 5796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb37bbea994428805c5d63246b62e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39675 } }
I20260927 15:02:36.347769 5791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:36.348934 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abb37bbea994428805c5d63246b62e8. Latest consensus state: current_term: 1 leader_uuid: "2abb37bbea994428805c5d63246b62e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb37bbea994428805c5d63246b62e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39675 } } }
I20260927 15:02:36.349064 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:36.349427 5805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:36.349354 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2abb37bbea994428805c5d63246b62e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb37bbea994428805c5d63246b62e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39675 } } }
I20260927 15:02:36.349836 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:36.351974 5805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:36.357259 5805 catalog_manager.cc:1347] Generated new cluster ID: 11017120be814f8cb48a9316bf3cff26
I20260927 15:02:36.357347 5805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:36.368453 5805 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:36.369419 5805 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:36.375810 5805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2abb37bbea994428805c5d63246b62e8: Generated new TSK 0
I20260927 15:02:36.376513 5805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:02:36.391803 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:36.386799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.98.62:39675)
0927 15:02:36.386929 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:36.386934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:36.387116 (+ 182us) server_negotiation.cc:407] Connection header received
0927 15:02:36.387264 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:36.387266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:36.387313 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:36.387418 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:02:36.387676 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.388203 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.389501 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.389720 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.389897 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.390194 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:02:36.391382 (+ 1188us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:02:36.391652 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:02:36.391877 5820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:36.386734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39675 (local address 127.0.0.1:53454)
0927 15:02:36.387019 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:36.387035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:36.387149 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:36.387436 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:36.387439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:36.387447 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:02:36.387565 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:36.387571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.388342 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.388346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:36.389355 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:36.389366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.389900 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.389921 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:36.390011 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.390039 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:02:36.391560 (+ 1521us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:02:36.391699 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
W20260927 15:02:36.391883 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 15:02:36.392544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5724
E20260927 15:02:36.402979 5821 webserver.cc:533] Webserver: sslize failed:
E20260927 15:02:36.403084 5821 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T15:02:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T15:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:36Z Disabled control of system clock
I20260927 15:02:36.440826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32965
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32965
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/testchainca.cert with env {}
W20260927 15:02:36.555549 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:36.556031 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:36.556185 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:36.560808 5825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:36.560875 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:36.560894 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:36.560958 5825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:36.560997 5825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:02:36.561038 5825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:02:36.561072 5825 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 15:02:36.566881 5825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:36.567935 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:36.568949 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:36.577584 5825 server_base.cc:1034] running on GCE node
W20260927 15:02:36.577540 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:36.577621 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:36.577548 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:36.578308 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:36.579356 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:36.580513 5825 hybrid_clock.cc:648] HybridClock initialized: now 1790521356580485 us; error 41 us; skew 500 ppm
I20260927 15:02:36.582294 5825 webserver.cc:460] Webserver started at http://127.0.98.62:46359/ using document root <none> and password file <none>
I20260927 15:02:36.582839 5825 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:36.582907 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:36.583134 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:36.585070 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "706a5546468e44f4a31f65b65348a778"
format_stamp: "Formatted at 2026-09-27 15:02:36 on dist-test-slave-ml7t"
server_key: "6c698ae3d8cd1b3aa594896c91389271"
server_key_iv: "043f2acb3afc7ef19f9781e3873ffb65"
server_key_version: "encryptionkey@0"
I20260927 15:02:36.585640 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "706a5546468e44f4a31f65b65348a778"
format_stamp: "Formatted at 2026-09-27 15:02:36 on dist-test-slave-ml7t"
server_key: "6c698ae3d8cd1b3aa594896c91389271"
server_key_iv: "043f2acb3afc7ef19f9781e3873ffb65"
server_key_version: "encryptionkey@0"
I20260927 15:02:36.589133 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:02:36.591616 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:36.592808 5825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:36.592927 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "706a5546468e44f4a31f65b65348a778"
format_stamp: "Formatted at 2026-09-27 15:02:36 on dist-test-slave-ml7t"
server_key: "6c698ae3d8cd1b3aa594896c91389271"
server_key_iv: "043f2acb3afc7ef19f9781e3873ffb65"
server_key_version: "encryptionkey@0"
I20260927 15:02:36.593026 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:36.607051 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:36.607662 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:36.607846 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:36.614399 5825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32965
I20260927 15:02:36.614408 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32965 every 8 connection(s)
I20260927 15:02:36.615388 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:36.616691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5825
I20260927 15:02:36.616843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:36.617126 392 external_mini_cluster.cc:1428] Setting key 4643a0c9f2e731108fbea346bb12b85b
I20260927 15:02:36.618563 5892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:36.623878 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778: Bootstrap starting.
I20260927 15:02:36.626165 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:36.618486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.0.98.62:32965)
0927 15:02:36.619119 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:36.619134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:36.619170 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:02:36.619820 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:36.619836 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:36.620230 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:36.620566 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:36.620976 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.621995 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.622897 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.623371 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.623544 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.624439 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:36.624444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:36.624456 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:02:36.624832 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:36.624885 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:02:36.624903 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:36.625012 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:36.625157 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 15:02:36.625421 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":144,"threads_started":1}
I20260927 15:02:36.626175 5892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:36.626955 5892 log.cc:824] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:36.628984 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:02:36.629045 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778: No bootstrap required, opened a new log
I20260927 15:02:36.632596 5892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a5546468e44f4a31f65b65348a778" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32965 } }
I20260927 15:02:36.632844 5892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:36.632915 5892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706a5546468e44f4a31f65b65348a778, State: Initialized, Role: FOLLOWER
I20260927 15:02:36.633409 5892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a5546468e44f4a31f65b65348a778" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32965 } }
I20260927 15:02:36.633571 5892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:36.633635 5892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:36.633733 5892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:36.634627 5892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a5546468e44f4a31f65b65348a778" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32965 } }
I20260927 15:02:36.634966 5892 leader_election.cc:302] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 706a5546468e44f4a31f65b65348a778; no voters:
I20260927 15:02:36.635300 5892 leader_election.cc:288] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:36.635427 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:36.635699 5897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [term 1 LEADER]: Becoming Leader. State: Replica: 706a5546468e44f4a31f65b65348a778, State: Running, Role: LEADER
I20260927 15:02:36.636018 5897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a5546468e44f4a31f65b65348a778" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32965 } }
I20260927 15:02:36.636502 5892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:36.637925 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 706a5546468e44f4a31f65b65348a778. Latest consensus state: current_term: 1 leader_uuid: "706a5546468e44f4a31f65b65348a778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a5546468e44f4a31f65b65348a778" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32965 } } }
I20260927 15:02:36.637910 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "706a5546468e44f4a31f65b65348a778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a5546468e44f4a31f65b65348a778" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32965 } } }
I20260927 15:02:36.638111 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:36.638191 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:36.638616 5911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:36.642215 5911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:36.648072 5911 catalog_manager.cc:1347] Generated new cluster ID: 0882c0e04b2a4cdbb306c64bfa3fbad1
I20260927 15:02:36.648177 5911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:36.658692 5911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:36.659627 5911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:36.667183 5911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 706a5546468e44f4a31f65b65348a778: Generated new TSK 0
I20260927 15:02:36.667873 5911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:02:36.785408 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:36.679576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.0.98.62:32965)
0927 15:02:36.679713 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:36.679718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:36.679923 (+ 205us) server_negotiation.cc:407] Connection header received
0927 15:02:36.680079 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:36.680082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:36.680127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:36.680235 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:02:36.680488 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.681034 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.682271 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.682470 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.682546 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.682864 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:02:36.785005 (+102141us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:02:36.785275 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:02:36.785504 5895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 15:02:36.785955 5921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:36.679520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32965 (local address 127.0.0.1:44802)
0927 15:02:36.679823 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:36.679839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:36.679960 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:36.680257 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:36.680260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:36.680268 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:02:36.680383 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:36.680389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.681149 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.681153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:36.682159 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:36.682170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:36.682566 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:36.682591 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:36.682679 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:36.682708 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:02:36.785258 (+102550us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:02:36.785459 (+ 201us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":115,"threads_started":1}
I20260927 15:02:36.786525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5825
2026-09-27T15:02:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (373 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:37 dist-test-slave-ml7t krb5kdc[5933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:37 dist-test-slave-ml7t krb5kdc[5933](info): set up 2 sockets
Sep 27 15:02:37 dist-test-slave-ml7t krb5kdc[5933](info): commencing operation
krb5kdc: starting...
W20260927 15:02:38.070197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:38 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:38.192397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:38.301060 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:38.301329 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:38.301371 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:38.305455 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:38.305534 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:38.305554 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:38.305572 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:38.305588 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:38.310972 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:38.312005 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:38.312963 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:38.321554 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:38.321734 5946 server_base.cc:1034] running on GCE node
W20260927 15:02:38.321593 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:38.321593 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:38.322499 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:38.323598 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:38.324780 5946 hybrid_clock.cc:648] HybridClock initialized: now 1790521358324767 us; error 29 us; skew 500 ppm
Sep 27 15:02:38 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:38.327705 5946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:38.328531 5946 webserver.cc:460] Webserver started at http://127.0.98.62:43177/ using document root <none> and password file <none>
I20260927 15:02:38.329083 5946 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:38.329142 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:38.329314 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:38.331353 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "42e473eca3d84bfa852a8b874bfb7cc6"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "2bd2f697b0ab96bd4daea3e12752b95c"
server_key_iv: "4b150e57225579ab5c180f91b3931445"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.331881 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "42e473eca3d84bfa852a8b874bfb7cc6"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "2bd2f697b0ab96bd4daea3e12752b95c"
server_key_iv: "4b150e57225579ab5c180f91b3931445"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.336493 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:02:38.339226 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:38.340498 5946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:38.340698 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "42e473eca3d84bfa852a8b874bfb7cc6"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "2bd2f697b0ab96bd4daea3e12752b95c"
server_key_iv: "4b150e57225579ab5c180f91b3931445"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.340814 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:38.357828 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:38.360664 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:38.360862 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:38.367918 5946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45865
I20260927 15:02:38.368216 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45865 every 8 connection(s)
I20260927 15:02:38.368947 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:38.371796 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:38.377486 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6: Bootstrap starting.
I20260927 15:02:38.378593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 5946
I20260927 15:02:38.378782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:38.379068 392 external_mini_cluster.cc:1428] Setting key 01f8dcbd9a81bc97678489cb0d789376
I20260927 15:02:38.380496 6014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:38.381420 6014 log.cc:824] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:38.383586 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6: No bootstrap required, opened a new log
I20260927 15:02:38.387868 6014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 15:02:38.388085 6014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:38.388120 6014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e473eca3d84bfa852a8b874bfb7cc6, State: Initialized, Role: FOLLOWER
Sep 27 15:02:38 dist-test-slave-ml7t krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:38.388669 6014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 15:02:38.388821 6014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:38.388900 6014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:38.389043 6014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:38.390349 6014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 15:02:38.390758 6014 leader_election.cc:302] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42e473eca3d84bfa852a8b874bfb7cc6; no voters:
I20260927 15:02:38.391098 6014 leader_election.cc:288] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:38.391222 6019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:38.391472 6019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 42e473eca3d84bfa852a8b874bfb7cc6, State: Running, Role: LEADER
I20260927 15:02:38.391836 6019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 15:02:38.392402 6014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:38.393656 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e473eca3d84bfa852a8b874bfb7cc6. Latest consensus state: current_term: 1 leader_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } } }
I20260927 15:02:38.393814 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:38.394204 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e473eca3d84bfa852a8b874bfb7cc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } } }
I20260927 15:02:38.394361 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:38.395392 6028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:38.395967 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:38.380332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.0.98.62:45865)
0927 15:02:38.380835 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:38.380854 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:38.380897 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:02:38.381801 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:38.381827 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:38.382205 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:38.382568 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:38.383283 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.384738 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.385588 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.386340 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.386492 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:38.389473 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:38.389479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:38.389495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:38.389525 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:38.391619 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:38.392128 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:38.392134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:38.392139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:38.392276 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:38.392622 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:38.392625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:38.392627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:38.393162 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:38.393366 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:38.394240 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 15:02:38.394562 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":163,"threads_started":1}
I20260927 15:02:38.398813 6028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:38.404026 6028 catalog_manager.cc:1347] Generated new cluster ID: 7b6b9eecd3e947f59d781d2bcb450e71
I20260927 15:02:38.404098 6028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:38.442677 6028 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:38.443955 6028 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:38.449607 6028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42e473eca3d84bfa852a8b874bfb7cc6: Generated new TSK 0
I20260927 15:02:38.450371 6028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:38.485616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:38.604051 6040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:38.604327 6040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:38.604370 6040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:38.608367 6040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:38.608433 6040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:38.608491 6040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:38.613752 6040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:38.614784 6040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:38.615772 6040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:38.625522 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:38.625517 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:38.625694 6040 server_base.cc:1034] running on GCE node
W20260927 15:02:38.625511 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:38.626196 6040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:38.626849 6040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:38.628003 6040 hybrid_clock.cc:648] HybridClock initialized: now 1790521358627989 us; error 27 us; skew 500 ppm
Sep 27 15:02:38 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:38.630749 6040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:38.631564 6040 webserver.cc:460] Webserver started at http://127.0.98.1:42837/ using document root <none> and password file <none>
I20260927 15:02:38.632130 6040 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:38.632189 6040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:38.632350 6040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:38.634366 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "78574109e2734d23987b59c5dca69b92"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "b7f0d76c0d31c1e735feae4960d7b73c"
server_key_iv: "1de5ef2719a2272c5bf22c7b17bd3b8d"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.634891 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "78574109e2734d23987b59c5dca69b92"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "b7f0d76c0d31c1e735feae4960d7b73c"
server_key_iv: "1de5ef2719a2272c5bf22c7b17bd3b8d"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.638801 6040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:02:38.641208 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:38.642562 6040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:38.642701 6040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "78574109e2734d23987b59c5dca69b92"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "b7f0d76c0d31c1e735feae4960d7b73c"
server_key_iv: "1de5ef2719a2272c5bf22c7b17bd3b8d"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.642787 6040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:38.699820 6040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:38.703531 6040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:38.703697 6040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:38.704322 6040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:38.705336 6040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:38.705389 6040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:38.705430 6040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:38.705453 6040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:38.714731 6040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40723
I20260927 15:02:38.714774 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40723 every 8 connection(s)
I20260927 15:02:38.715623 6040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:38.723412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6040
I20260927 15:02:38.723538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:38.723809 392 external_mini_cluster.cc:1428] Setting key 9ddafd46271bebcd1fd484634afd9d16
Sep 27 15:02:38 dist-test-slave-ml7t krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:38.731071 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:38.717550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52783 (local address 127.0.98.62:45865)
0927 15:02:38.717725 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:38.717729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:38.718584 (+ 855us) server_negotiation.cc:407] Connection header received
0927 15:02:38.719992 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:38.719995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:38.720049 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:38.720150 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:38.721291 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.721884 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.723002 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.723324 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.723443 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:38.726611 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:38.726614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:38.726616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:38.726642 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:38.728498 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:38.729054 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:38.729060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:38.729065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:38.729169 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:38.729785 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:38.729792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:38.729795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:38.730057 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:38.730180 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:38.730666 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:02:38.730819 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:02:38.731755 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:38.717934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.1:52783)
0927 15:02:38.718434 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:38.718475 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:38.719780 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:38.720331 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:38.720337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:38.720684 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:38.721135 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:38.721157 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.722001 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.722005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:38.722875 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:38.722884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.723531 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.723589 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:38.723795 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:38.724493 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:38.724517 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:38.726384 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:38.728673 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:38.728678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:38.728690 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:38.728942 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:38.729281 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:38.729284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:38.729285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:38.729612 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:38.730179 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:38.730183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:38.730408 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 15:02:38.730569 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:02:38.730847 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":156,"threads_started":1}
I20260927 15:02:38.733137 6169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260927 15:02:38.733424 6169 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:02:38.734093 6169 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260927 15:02:38.735841 5978 ts_manager.cc:194] Registered new tserver with Master: 78574109e2734d23987b59c5dca69b92 (127.0.98.1:40723)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:38.737291 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:38.752533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:38.863227 6174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:38.863481 6174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:38.863523 6174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:38.867646 6174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:38.867713 6174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:38.867806 6174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:38.873018 6174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:38.874152 6174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:38.875185 6174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:38.884904 6180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:38.884907 6182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:38.884907 6179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:38.884955 6174 server_base.cc:1034] running on GCE node
I20260927 15:02:38.885574 6174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:38.886262 6174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:38.887424 6174 hybrid_clock.cc:648] HybridClock initialized: now 1790521358887409 us; error 25 us; skew 500 ppm
Sep 27 15:02:38 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:38.890307 6174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:38.891170 6174 webserver.cc:460] Webserver started at http://127.0.98.2:44155/ using document root <none> and password file <none>
I20260927 15:02:38.891736 6174 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:38.891795 6174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:38.892033 6174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:38.894515 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "3120d2592c1740358cdbf3a7a0eda1f8"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "3cf382386ea45141eb27c2d6af6ef337"
server_key_iv: "b5c72e180ed23a4101dc0cfc8a71a866"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.895134 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3120d2592c1740358cdbf3a7a0eda1f8"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "3cf382386ea45141eb27c2d6af6ef337"
server_key_iv: "b5c72e180ed23a4101dc0cfc8a71a866"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.898797 6174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:02:38.901091 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:38.902348 6174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:38.902464 6174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "3120d2592c1740358cdbf3a7a0eda1f8"
format_stamp: "Formatted at 2026-09-27 15:02:38 on dist-test-slave-ml7t"
server_key: "3cf382386ea45141eb27c2d6af6ef337"
server_key_iv: "b5c72e180ed23a4101dc0cfc8a71a866"
server_key_version: "encryptionkey@0"
I20260927 15:02:38.902563 6174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:38.917018 6174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:38.920289 6174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:38.920504 6174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:38.921141 6174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:38.922097 6174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:38.922147 6174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:38.922215 6174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:38.922256 6174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:38.931414 6174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39739
I20260927 15:02:38.931432 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39739 every 8 connection(s)
I20260927 15:02:38.932368 6174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:38.939025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6174
I20260927 15:02:38.939155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:38.939447 392 external_mini_cluster.cc:1428] Setting key 16d9a812448e7b6bc10de8fc8544d91d
Sep 27 15:02:38 dist-test-slave-ml7t krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:38.946636 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:38.934239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34341 (local address 127.0.98.62:45865)
0927 15:02:38.934371 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:38.934374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:38.935220 (+ 846us) server_negotiation.cc:407] Connection header received
0927 15:02:38.936613 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:38.936616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:38.936675 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:38.936781 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:38.937977 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.938508 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.939782 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.940001 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.940116 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:38.942735 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:38.942738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:38.942740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:38.942781 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:38.944297 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:38.944921 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:38.944924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:38.944925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:38.945002 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:38.945367 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:38.945371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:38.945373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:38.945575 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:38.945681 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:38.946356 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 15:02:38.946472 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:02:38.947602 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:38.934633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.2:34341)
0927 15:02:38.935068 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:38.935111 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:38.936392 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:38.936955 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:38.936962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:38.937310 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:38.937842 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:38.937858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.938642 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.938645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:38.939661 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:38.939670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:38.940151 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:38.940155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:38.940331 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:38.940933 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:38.940948 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:38.942556 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:38.944408 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:38.944417 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:38.944435 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:38.944827 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:38.945088 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:38.945090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:38.945091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:38.945253 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:38.945689 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:38.945698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:38.946074 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 15:02:38.946304 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 15:02:38.946621 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:38.948814 6303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260927 15:02:38.949064 6303 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:38.949597 6303 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260927 15:02:38.950716 5978 ts_manager.cc:194] Registered new tserver with Master: 3120d2592c1740358cdbf3a7a0eda1f8 (127.0.98.2:39739)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:38.951391 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:38.964915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:39.074859 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:39.075189 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:39.075281 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:39.079478 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:39.079578 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:39.079681 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:39.085067 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:39.086189 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:39.087237 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:39.097216 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:39.097199 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:39.097301 6308 server_base.cc:1034] running on GCE node
W20260927 15:02:39.097201 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:39.097968 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:39.098678 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:39.099858 6308 hybrid_clock.cc:648] HybridClock initialized: now 1790521359099822 us; error 50 us; skew 500 ppm
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.102730 6308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:39.103616 6308 webserver.cc:460] Webserver started at http://127.0.98.3:41653/ using document root <none> and password file <none>
I20260927 15:02:39.104223 6308 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:39.104315 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:39.104529 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:39.106736 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "adfd30bb2204472db2ec3d270723dc35"
format_stamp: "Formatted at 2026-09-27 15:02:39 on dist-test-slave-ml7t"
server_key: "4d75c139176ac60eeaaf203b171f953e"
server_key_iv: "e65efde46f3a0229fc34db62c331f0a3"
server_key_version: "encryptionkey@0"
I20260927 15:02:39.107299 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "adfd30bb2204472db2ec3d270723dc35"
format_stamp: "Formatted at 2026-09-27 15:02:39 on dist-test-slave-ml7t"
server_key: "4d75c139176ac60eeaaf203b171f953e"
server_key_iv: "e65efde46f3a0229fc34db62c331f0a3"
server_key_version: "encryptionkey@0"
I20260927 15:02:39.111042 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:39.113554 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:39.114759 6308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:39.114894 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "adfd30bb2204472db2ec3d270723dc35"
format_stamp: "Formatted at 2026-09-27 15:02:39 on dist-test-slave-ml7t"
server_key: "4d75c139176ac60eeaaf203b171f953e"
server_key_iv: "e65efde46f3a0229fc34db62c331f0a3"
server_key_version: "encryptionkey@0"
I20260927 15:02:39.115003 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:39.140300 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:39.143390 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:39.143599 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:39.144320 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:39.145227 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:39.145309 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:39.145375 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:39.145426 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:39.155154 6308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46567
I20260927 15:02:39.155181 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46567 every 8 connection(s)
I20260927 15:02:39.156091 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:39.161087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6308
I20260927 15:02:39.161192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:39.161478 392 external_mini_cluster.cc:1428] Setting key 675feb133d40ec24c0850a113d35bf14
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:39.170164 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:39.158081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33051 (local address 127.0.98.62:45865)
0927 15:02:39.158231 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:39.158236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:39.159189 (+ 953us) server_negotiation.cc:407] Connection header received
0927 15:02:39.160526 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:39.160529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:39.160575 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:39.160670 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:39.162064 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.162828 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.164082 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.164348 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.164458 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:39.166854 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:39.166857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:39.166859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:39.166883 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:39.168372 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.168799 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.168801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.168802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.168871 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.169167 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.169169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.169171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.169339 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:39.169411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:39.169936 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 15:02:39.170037 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:02:39.171026 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:39.158570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.3:33051)
0927 15:02:39.159030 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:39.159072 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:39.160314 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:39.160817 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:39.160823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:39.161228 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:39.161876 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:39.161898 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.162985 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.162988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:39.163994 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:39.164003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.164499 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.164502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:39.164684 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:39.165274 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:39.165301 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:39.166704 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:39.168464 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:39.168468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:39.168479 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:39.168715 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:39.168948 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:39.168950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:39.168951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:39.169093 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:39.169422 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:39.169427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:39.169643 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 15:02:39.169884 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 15:02:39.170131 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260927 15:02:39.172251 6437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260927 15:02:39.172499 6437 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:39.173008 6437 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260927 15:02:39.174163 5978 ts_manager.cc:194] Registered new tserver with Master: adfd30bb2204472db2ec3d270723dc35 (127.0.98.3:46567)
I20260927 15:02:39.174787 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33051
I20260927 15:02:39.175418 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/test-admin.keytab.
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.190972 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 15:02:39.193924 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:39.192350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.0.98.1:40723)
0927 15:02:39.192713 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:39.192728 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:39.192750 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:02:39.192798 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:39.192802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:39.192983 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:39.193136 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:39.193685 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54334: BlockingRecv error: recv got EOF from 127.0.0.1:54334 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":162,"threads_started":1}
W20260927 15:02:39.194205 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc)
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.195253 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.196169 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:39.205240 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:39.197065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.0.98.1:40723)
0927 15:02:39.197191 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:39.197195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:39.197219 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:02:39.197323 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:39.197325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:39.197362 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:39.197429 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:39.197964 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.198725 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.199499 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.199859 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.199939 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:39.202141 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:39.202149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:39.202163 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:39.202185 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:39.203682 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.204134 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.204139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.204140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.204214 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.204440 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.204442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.204443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.204671 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:39.204847 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:39.204980 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 15:02:39.205097 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 15:02:39.209159 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:39.208320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.0.98.1:40723)
0927 15:02:39.208452 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:39.208455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:39.208487 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:02:39.208598 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:39.208600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:39.208634 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:39.208698 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:39.209063 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54364: BlockingRecv error: recv got EOF from 127.0.0.1:54364 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:02:39.209278 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc)
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.210422 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.211303 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:39.219645 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:39.212120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.0.98.1:40723)
0927 15:02:39.212254 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:39.212257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:39.212269 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:39.212402 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:39.212404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:39.212442 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:39.212512 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:39.212979 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.213510 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.214322 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.214547 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.214626 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:39.216731 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:39.216734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:39.216736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:39.216757 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:39.218069 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.218445 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.218448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.218449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.218519 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.218816 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.218819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.218821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.218989 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:39.219126 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:39.219366 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 15:02:39.219504 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 15:02:39.222213 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:39.221321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.0.98.1:40723)
0927 15:02:39.221450 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:39.221453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:39.221479 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:02:39.221584 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:39.221586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:39.221619 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:39.221680 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:39.222116 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54382: BlockingRecv error: recv got EOF from 127.0.0.1:54382 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:02:39.222260 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521328452215-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.223228 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:39.224063 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:02:39.231796 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:39.224796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.0.98.1:40723)
0927 15:02:39.224937 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:39.224940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:39.225014 (+ 74us) server_negotiation.cc:407] Connection header received
0927 15:02:39.225098 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:39.225100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:39.225131 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:39.225194 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:39.225629 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.226163 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.226877 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:39.227084 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:39.227157 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:39.229149 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:39.229152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:39.229154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:39.229173 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:39.230447 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.230801 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.230804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.230805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.230869 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:39.231082 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:39.231084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:39.231085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:39.231218 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:39.231335 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:39.231520 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:02:39.231626 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:02:39.232764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6040
I20260927 15:02:39.238255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6174
I20260927 15:02:39.243497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6308
I20260927 15:02:39.250497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 5946
2026-09-27T15:02:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2222 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[6457](info): set up 2 sockets
Sep 27 15:02:39 dist-test-slave-ml7t krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260927 15:02:40.302206 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:40 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 15:02:40.425169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:40.536530 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:40.536877 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:40.536970 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:40.541148 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:40.541247 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:40.541287 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:40.541325 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:40.541368 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:40.546749 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:40.547843 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:40.548877 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:40.557474 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:40.557543 6470 server_base.cc:1034] running on GCE node
W20260927 15:02:40.557467 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:40.557467 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:40.558288 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:40.559388 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:40.560547 6470 hybrid_clock.cc:648] HybridClock initialized: now 1790521360560528 us; error 31 us; skew 500 ppm
Sep 27 15:02:40 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:40.563371 6470 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 15:02:40.564320 6470 webserver.cc:460] Webserver started at http://127.0.98.62:43445/ using document root <none> and password file <none>
I20260927 15:02:40.564893 6470 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:40.564955 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:40.565171 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:40.567310 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8"
format_stamp: "Formatted at 2026-09-27 15:02:40 on dist-test-slave-ml7t"
server_key: "9524ac8b0a193a5864da9b7b6b0a3bba"
server_key_iv: "ab591c45a8332cd929afc191e995d421"
server_key_version: "encryptionkey@0"
I20260927 15:02:40.567879 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8"
format_stamp: "Formatted at 2026-09-27 15:02:40 on dist-test-slave-ml7t"
server_key: "9524ac8b0a193a5864da9b7b6b0a3bba"
server_key_iv: "ab591c45a8332cd929afc191e995d421"
server_key_version: "encryptionkey@0"
I20260927 15:02:40.571666 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:02:40.574148 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:40.575343 6470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:40.575503 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8"
format_stamp: "Formatted at 2026-09-27 15:02:40 on dist-test-slave-ml7t"
server_key: "9524ac8b0a193a5864da9b7b6b0a3bba"
server_key_iv: "ab591c45a8332cd929afc191e995d421"
server_key_version: "encryptionkey@0"
I20260927 15:02:40.575621 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:40.602736 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:40.605798 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:40.606009 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:40.613092 6470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41127
I20260927 15:02:40.613391 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41127 every 8 connection(s)
I20260927 15:02:40.614123 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:40.616864 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:40.621505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6470
I20260927 15:02:40.621639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:40.621974 392 external_mini_cluster.cc:1428] Setting key bf0e86a1203310724ef0b15141201190
I20260927 15:02:40.622671 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: Bootstrap starting.
I20260927 15:02:40.625030 6538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:40.625833 6538 log.cc:824] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:40.627612 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: No bootstrap required, opened a new log
Sep 27 15:02:40 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:40.631703 6538 raft_consensus.cc:348] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260927 15:02:40.631994 6538 raft_consensus.cc:374] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:40.632076 6538 raft_consensus.cc:729] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf76d75dba64707a8ccd1ef2fa9a0c8, State: Initialized, Role: FOLLOWER
I20260927 15:02:40.632568 6538 consensus_queue.cc:260] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260927 15:02:40.632766 6538 raft_consensus.cc:388] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:40.632839 6538 raft_consensus.cc:482] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:40.632946 6538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:40.634105 6538 raft_consensus.cc:504] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260927 15:02:40.634479 6538 leader_election.cc:302] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daf76d75dba64707a8ccd1ef2fa9a0c8; no voters:
I20260927 15:02:40.634827 6538 leader_election.cc:288] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:40.634925 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:40.635166 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 1 LEADER]: Becoming Leader. State: Replica: daf76d75dba64707a8ccd1ef2fa9a0c8, State: Running, Role: LEADER
I20260927 15:02:40.635538 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } }
I20260927 15:02:40.636062 6538 sys_catalog.cc:564] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:40.636720 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daf76d75dba64707a8ccd1ef2fa9a0c8. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } }
I20260927 15:02:40.636720 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } }
I20260927 15:02:40.636827 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:40.623372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.0.98.62:41127)
0927 15:02:40.623816 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:40.623831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:40.623873 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:02:40.624498 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:40.624515 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:40.624858 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:40.625191 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:40.626114 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:40.627063 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:40.628039 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:40.628705 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:40.628864 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:40.631431 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:40.631437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:40.631453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:40.631483 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:40.633642 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:40.634101 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:40.634107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:40.634114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:40.634262 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:40.634520 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:40.634523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:40.634525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:40.635036 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:40.635226 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:40.635531 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 15:02:40.635852 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260927 15:02:40.636937 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:40.636935 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:40.637652 6552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:40.640663 6552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:40.646428 6552 catalog_manager.cc:1347] Generated new cluster ID: 4911026bf1eb4171bc627eb4ce0f4c46
I20260927 15:02:40.646500 6552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:40.661583 6552 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:40.662926 6552 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:40.670946 6552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: Generated new TSK 0
I20260927 15:02:40.671811 6552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 15:02:40.704708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:40.815107 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:40.815387 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:40.815433 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:40.819571 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:40.819633 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:40.819741 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:40.825656 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:40.826783 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:40.827797 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:40.837422 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:40.837561 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:40.837882 6564 server_base.cc:1034] running on GCE node
W20260927 15:02:40.837486 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:40.838289 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:40.838908 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:40.840090 6564 hybrid_clock.cc:648] HybridClock initialized: now 1790521360840076 us; error 40 us; skew 500 ppm
Sep 27 15:02:40 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:40.842816 6564 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 15:02:40.843645 6564 webserver.cc:460] Webserver started at http://127.0.98.1:42817/ using document root <none> and password file <none>
I20260927 15:02:40.844197 6564 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:40.844288 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:40.844520 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:40.846606 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2"
format_stamp: "Formatted at 2026-09-27 15:02:40 on dist-test-slave-ml7t"
server_key: "ec7708502d0a4d3104c6a5544459e543"
server_key_iv: "afefd4c7dc25b7d985dd1d90960e6061"
server_key_version: "encryptionkey@0"
I20260927 15:02:40.847157 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2"
format_stamp: "Formatted at 2026-09-27 15:02:40 on dist-test-slave-ml7t"
server_key: "ec7708502d0a4d3104c6a5544459e543"
server_key_iv: "afefd4c7dc25b7d985dd1d90960e6061"
server_key_version: "encryptionkey@0"
I20260927 15:02:40.850841 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:40.853264 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:40.854486 6564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:40.854619 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2"
format_stamp: "Formatted at 2026-09-27 15:02:40 on dist-test-slave-ml7t"
server_key: "ec7708502d0a4d3104c6a5544459e543"
server_key_iv: "afefd4c7dc25b7d985dd1d90960e6061"
server_key_version: "encryptionkey@0"
I20260927 15:02:40.854733 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:40.868844 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:40.872220 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:40.872426 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:40.873013 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:40.873853 6564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:40.873926 6564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:40.873993 6564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:40.874044 6564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:40.882860 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38235
I20260927 15:02:40.882889 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38235 every 8 connection(s)
I20260927 15:02:40.883766 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:40.890741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6564
I20260927 15:02:40.890880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:40.891176 392 external_mini_cluster.cc:1428] Setting key c65d227a0720671b2eec8f7e6e73cf69
Sep 27 15:02:40 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:40.897507 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:40.885597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59811 (local address 127.0.98.62:41127)
0927 15:02:40.885722 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:40.885726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:40.886582 (+ 856us) server_negotiation.cc:407] Connection header received
0927 15:02:40.887801 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:40.887804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:40.887849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:40.887922 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:40.889008 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:40.889542 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:40.890781 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:40.891103 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:40.891181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:40.893904 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:40.893907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:40.893909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:40.893930 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:40.895349 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:40.895964 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:40.895967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:40.895968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:40.896037 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:40.896450 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:40.896453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:40.896455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:40.896624 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:40.896699 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:40.897307 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:02:40.897396 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:02:40.898772 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:40.885989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:59811)
0927 15:02:40.886428 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:40.886469 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:40.887614 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:40.888073 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:40.888079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:40.888427 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:40.888873 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:40.888885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:40.889645 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:40.889648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:40.890657 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:40.890668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:40.891268 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:40.891273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:40.891461 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:40.892097 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:40.892113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:40.893718 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:40.895463 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:40.895471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:40.895487 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:40.895853 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:40.896125 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:40.896131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:40.896136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:40.896328 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:40.896735 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:40.896742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:40.897052 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 15:02:40.897266 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 15:02:40.897625 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260927 15:02:40.899987 6693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260927 15:02:40.900228 6693 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:02:40.900703 6693 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260927 15:02:40.902207 6502 ts_manager.cc:194] Registered new tserver with Master: e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1:38235)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:40.903431 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 15:02:40.918474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:41.030079 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:41.030692 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:41.030820 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:41.035390 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:41.035480 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:41.035578 6698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:41.041376 6698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:41.042601 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:41.043641 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:41.053580 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:41.053694 6698 server_base.cc:1034] running on GCE node
W20260927 15:02:41.053639 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.053643 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:41.054288 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:41.054925 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:41.056092 6698 hybrid_clock.cc:648] HybridClock initialized: now 1790521361056073 us; error 32 us; skew 500 ppm
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:41.065145 6698 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 15:02:41.066174 6698 webserver.cc:460] Webserver started at http://127.0.98.2:43405/ using document root <none> and password file <none>
I20260927 15:02:41.066743 6698 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:41.066805 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:41.066975 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:41.069034 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee668e13116e4035b8861fd5af19cfa3"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "6ecba537353ea46d53b7f04b593e2551"
server_key_iv: "6d7a4419d9ecc8d350ece2e71aacc676"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.069584 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee668e13116e4035b8861fd5af19cfa3"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "6ecba537353ea46d53b7f04b593e2551"
server_key_iv: "6d7a4419d9ecc8d350ece2e71aacc676"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.073920 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:02:41.076411 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.077737 6698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:41.077931 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "ee668e13116e4035b8861fd5af19cfa3"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "6ecba537353ea46d53b7f04b593e2551"
server_key_iv: "6d7a4419d9ecc8d350ece2e71aacc676"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.078033 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:41.097160 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:41.100180 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:41.100337 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:41.100871 6698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:41.101639 6698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:41.101693 6698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.101732 6698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:41.101747 6698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.111018 6698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34969
I20260927 15:02:41.111042 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34969 every 8 connection(s)
I20260927 15:02:41.111904 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:41.114897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6698
I20260927 15:02:41.115017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:41.115373 392 external_mini_cluster.cc:1428] Setting key 44e18f1d1f148e47799dda6173140f7b
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:41.127619 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.113743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43401 (local address 127.0.98.62:41127)
0927 15:02:41.113901 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.113904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.114853 (+ 949us) server_negotiation.cc:407] Connection header received
0927 15:02:41.116428 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.116431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.116483 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.116565 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.118268 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.118783 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.120362 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.120781 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.120902 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.123464 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.123468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.123472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.123503 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.125341 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.125912 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.125918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.125920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.126035 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.126358 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.126361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.126362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.126592 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.126716 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.127298 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.127427 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:41.128495 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.114091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:43401)
0927 15:02:41.114664 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.114726 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.116215 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.116746 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.116753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.117310 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.118088 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.118112 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.118912 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.118918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.120228 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.120237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.120934 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.120936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.121132 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.121859 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.121874 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.123282 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.125458 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.125462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.125474 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.125762 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.126134 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.126136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.126137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.126264 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.126716 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.126723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.127029 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.127237 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.127596 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":116,"threads_started":1}
I20260927 15:02:41.130045 6827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260927 15:02:41.130340 6827 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:41.130970 6827 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260927 15:02:41.132051 6502 ts_manager.cc:194] Registered new tserver with Master: ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969)
I20260927 15:02:41.132627 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 15:02:41.144057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:41.268939 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:41.269412 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:41.269521 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:41.276578 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:41.276680 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:41.276821 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:41.287465 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:41.289183 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:41.290735 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:41.302058 6832 server_base.cc:1034] running on GCE node
W20260927 15:02:41.302039 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.302062 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.302078 6838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:41.302752 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:41.303443 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:41.304633 6832 hybrid_clock.cc:648] HybridClock initialized: now 1790521361304621 us; error 32 us; skew 500 ppm
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:41.307407 6832 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 15:02:41.308240 6832 webserver.cc:460] Webserver started at http://127.0.98.3:41833/ using document root <none> and password file <none>
I20260927 15:02:41.308807 6832 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:41.308903 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:41.309144 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:41.311183 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef32965fdc614ddb9312d28c310ba524"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "4dde937574dc3d8a9559c4e93e9a76aa"
server_key_iv: "57629a19d5a3898a6320f7ebed300a0b"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.311793 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef32965fdc614ddb9312d28c310ba524"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "4dde937574dc3d8a9559c4e93e9a76aa"
server_key_iv: "57629a19d5a3898a6320f7ebed300a0b"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.315534 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:02:41.317989 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.319358 6832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:41.319497 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "ef32965fdc614ddb9312d28c310ba524"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "4dde937574dc3d8a9559c4e93e9a76aa"
server_key_iv: "57629a19d5a3898a6320f7ebed300a0b"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.319617 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:41.338753 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:41.342103 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:41.342344 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:41.342957 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:41.343793 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:41.343870 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.343937 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:41.343986 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.354373 6832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46395
I20260927 15:02:41.354390 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46395 every 8 connection(s)
I20260927 15:02:41.355296 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:41.360800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6832
I20260927 15:02:41.360909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:41.361198 392 external_mini_cluster.cc:1428] Setting key 67f4b95f5ef617a0bf73eec314b05c80
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:41.369557 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.357184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54147 (local address 127.0.98.62:41127)
0927 15:02:41.357319 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.357324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.358147 (+ 823us) server_negotiation.cc:407] Connection header received
0927 15:02:41.359643 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.359646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.359703 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.359774 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.361033 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.361609 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.363471 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.363728 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.363811 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.366282 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.366285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.366287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.366311 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.367547 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.368055 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.368057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.368059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.368133 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.368511 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.368513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.368514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.368689 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.368767 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.369311 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.369430 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:02:41.370434 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.357474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:54147)
0927 15:02:41.357970 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.358015 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.359426 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.359948 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.359955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.360342 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.360838 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.360880 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.361762 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.361765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.363322 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.363333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.363858 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.363861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.364016 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.364571 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.364587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.366116 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.367676 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.367681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.367695 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.367956 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.368258 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.368260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.368262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.368415 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.368779 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.368784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.369034 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.369193 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.369484 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260927 15:02:41.371672 6961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260927 15:02:41.371950 6961 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:41.372457 6961 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260927 15:02:41.373478 6502 ts_manager.cc:194] Registered new tserver with Master: ef32965fdc614ddb9312d28c310ba524 (127.0.98.3:46395)
I20260927 15:02:41.374114 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54147
I20260927 15:02:41.375586 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 15:02:41.400892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35525
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41127,127.0.98.61:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:41.511886 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:41.512184 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:41.512251 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:41.516242 6966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:41.516355 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:41.516395 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:41.516434 6966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:41.516469 6966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:41.521786 6966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41127,127.0.98.61:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
W20260927 15:02:41.522075 6966 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41127,127.0.98.61:35525'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 15:02:41.523036 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:41.524148 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:41.532765 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.532765 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.532759 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:41.532817 6966 server_base.cc:1034] running on GCE node
I20260927 15:02:41.533645 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:41.534806 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:41.536026 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790521361536013 us; error 33 us; skew 500 ppm
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:41.539646 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 15:02:41.540589 6966 webserver.cc:460] Webserver started at http://127.0.98.61:42537/ using document root <none> and password file <none>
I20260927 15:02:41.541175 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:41.541271 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:41.541501 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:41.543668 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/instance:
uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "ca74d018d85e4bc2a598bca204d6dd0c"
server_key_iv: "cea60d42615b548b119872ea7cfe0813"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.544247 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal/instance:
uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "ca74d018d85e4bc2a598bca204d6dd0c"
server_key_iv: "cea60d42615b548b119872ea7cfe0813"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.548076 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:02:41.550623 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.551930 6966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:41.552075 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "ca74d018d85e4bc2a598bca204d6dd0c"
server_key_iv: "cea60d42615b548b119872ea7cfe0813"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.552196 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:41.570904 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:41.574321 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:41.574530 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:41.581516 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35525
I20260927 15:02:41.581856 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35525 every 8 connection(s)
I20260927 15:02:41.582683 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
I20260927 15:02:41.585426 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:41.587028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 6966
I20260927 15:02:41.589856 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:41.601395 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.591633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.98.62:41127)
0927 15:02:41.591772 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.591777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.592122 (+ 345us) server_negotiation.cc:407] Connection header received
0927 15:02:41.592249 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.592253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.592305 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.592406 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.593042 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.593580 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.594716 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.594924 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.595040 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.597752 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.597755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.597759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.597808 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.599446 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.599939 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.599943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.599945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.600025 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.600359 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.600364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.600366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.600576 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.600687 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.601124 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.601235 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:41.602360 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.588528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.0.98.61:35525)
0927 15:02:41.588923 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.588943 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.589150 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:02:41.589982 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.589999 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.590362 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.590686 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.591499 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.592798 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.593686 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.594288 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.594404 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.597818 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.597825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.597840 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.597866 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.599752 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.600167 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.600173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.600175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.600270 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.600554 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.600557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.600559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.600903 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.601099 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.601341 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.601485 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
I20260927 15:02:41.602360 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.591565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33300)
0927 15:02:41.591988 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.592037 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.592145 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.592451 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.592455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.592766 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.592916 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.592924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.593741 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.593743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.594594 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.594605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.595096 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.595100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.595193 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.595729 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.595745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.597601 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.599551 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.599558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.599566 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.599829 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.600119 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.600124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.600126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.600263 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.600683 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.600690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.600914 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.601082 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.601400 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":29,"spinlock_wait_cycles":1280,"thread_start_us":208,"threads_started":1}
I20260927 15:02:41.603737 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 15:02:41.611145 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.604157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.0.1:33694)
0927 15:02:41.604287 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.604301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.604370 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.604634 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.604637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.604798 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.604879 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.604884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.605935 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.605937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.606460 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.606467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.606848 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.606850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.606917 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.607309 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.607320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.608418 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.609891 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.609894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.609895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.610070 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.610310 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.610313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.610314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.610382 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.610764 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.610766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.610834 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.610916 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.611006 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 15:02:41.611255 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.604251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.0.98.61:35525)
0927 15:02:41.604339 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.604342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.604353 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:02:41.604451 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.604453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.604511 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.604594 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.604970 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.605849 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.606546 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.606759 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.606834 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.608526 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.608529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.608530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.608549 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.609749 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.610149 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.610151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.610152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.610234 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.610486 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.610489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.610490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.610675 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.610760 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.610969 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.611055 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:02:41.613974 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Bootstrap starting.
I20260927 15:02:41.616165 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:41.616990 7034 log.cc:824] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:41.618961 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: No bootstrap required, opened a new log
I20260927 15:02:41.621878 7034 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } }
I20260927 15:02:41.622100 7034 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:41.622155 7034 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e6f6795b0b4b968e95ca789b89be9a, State: Initialized, Role: FOLLOWER
I20260927 15:02:41.622661 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } }
I20260927 15:02:41.623132 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } } }
I20260927 15:02:41.623262 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:02:41.623869 7034 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:02:41.628156 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:02:41.628237 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:02:41.630175 6966 master_runner.cc:428] Detected that this master f2e6f6795b0b4b968e95ca789b89be9a is joining an existing cluster
I20260927 15:02:41.630249 6966 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35525
I20260927 15:02:41.640780 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.632532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.98.62:41127)
0927 15:02:41.632658 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.632663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.632815 (+ 152us) server_negotiation.cc:407] Connection header received
0927 15:02:41.632937 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.632941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.632990 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.633081 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.633642 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.634382 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.635362 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.635581 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.635696 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.637027 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.637031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.637034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.637087 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.638864 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.639469 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.639472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.639474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.639560 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.639891 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.639896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.639899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.640126 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.640246 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.640515 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.640625 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:02:41.640797 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.632442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33316)
0927 15:02:41.632710 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.632728 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.632832 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.633131 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.633134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.633396 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.633535 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.633541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.634514 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.634519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.635246 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.635255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.635709 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.635712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.635865 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.636444 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.636458 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.636904 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.638989 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.638992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.638994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.639352 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.639685 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.639689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.639692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.639794 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.640243 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.640246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.640353 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.640463 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.640595 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":50,"thread_start_us":103,"threads_started":1}
I20260927 15:02:41.641222 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.632919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.98.61:35525)
0927 15:02:41.633005 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.633009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.633433 (+ 424us) server_negotiation.cc:407] Connection header received
0927 15:02:41.633518 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.633521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.633561 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.633641 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.634133 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.634856 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.636377 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.636707 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.636811 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.638077 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.638083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.638086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.638111 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.639604 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.640091 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.640094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.640122 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.640206 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.640473 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.640476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.640478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.640639 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.640701 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.640969 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.641068 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 15:02:41.641635 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.633012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.0.1:33708)
0927 15:02:41.633328 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.633343 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.633441 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.633662 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.633664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.633903 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.634014 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.634020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.634971 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.634975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.635927 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.635937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.636826 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.636830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.636926 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.637455 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.637469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.637945 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.639737 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.639741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.639743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.639991 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.640304 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.640307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.640309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.640393 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.640755 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.640757 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.640829 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.641341 (+ 512us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.641470 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":21,"spinlock_wait_cycles":7296,"thread_start_us":77,"threads_started":1}
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 15:02:41.656107 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.646126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.0.98.61:35525)
0927 15:02:41.646248 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.646251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.647009 (+ 758us) server_negotiation.cc:407] Connection header received
0927 15:02:41.647211 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.647213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.647253 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.647319 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.648069 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.648549 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.649829 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.650037 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.650114 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.652627 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.652631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.652633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.652664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.654115 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.654606 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.654612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.654614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.654691 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.654978 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.654981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.654983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.655271 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.655364 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.655814 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.655932 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:02:41.656248 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.646104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.0.1:33716)
0927 15:02:41.646880 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.646934 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.647069 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.647378 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.647384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.647832 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.647962 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.647969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.648650 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.648653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.649669 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.649679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.650149 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.650153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.650264 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.650755 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.650769 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.652480 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.654246 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.654256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.654263 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.654503 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.654787 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.654790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.654792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.654880 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.655386 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.655393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.655577 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.655747 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.655960 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":80,"threads_started":1}
I20260927 15:02:41.658375 6501 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35525
}
attrs {
promote: true
}
}
dest_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8"
cas_config_opid_index: -1
I20260927 15:02:41.659130 6501 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } }
I20260927 15:02:41.660810 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } }
I20260927 15:02:41.661468 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:41.662578 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35525
I20260927 15:02:41.662823 7008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Refusing update from remote peer daf76d75dba64707a8ccd1ef2fa9a0c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 15:02:41.662978 6966 master.cc:537] Master@127.0.98.61:35525 shutting down...
I20260927 15:02:41.663314 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } }
I20260927 15:02:41.663408 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:41.664073 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:02:41.666189 6489 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 15:02:41.667187 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 -> Peer f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61:35525): Couldn't send request to peer f2e6f6795b0b4b968e95ca789b89be9a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 15:02:41.667886 6966 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:41.668177 6966 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:41.668306 6966 tablet_replica.cc:330] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 15:02:41.675114 6966 master.cc:552] Master@127.0.98.61:35525 shutdown complete.
I20260927 15:02:41.675173 6966 master_runner.cc:305] Clearing existing system tablet
I20260927 15:02:41.675411 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:41.675464 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:41.676789 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 15:02:41.677703 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.677959 6966 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:41.678081 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:41.678115 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "ca74d018d85e4bc2a598bca204d6dd0c"
server_key_iv: "cea60d42615b548b119872ea7cfe0813"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.680205 6966 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:41.683171 6966 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:02:41.683260 6966 log.cc:1196] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 15:02:41.683499 6966 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Deleting consensus metadata
I20260927 15:02:41.683669 6966 master_runner.cc:315] Copying system tablet from 127.0.98.62:41127
I20260927 15:02:41.684803 6966 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 15:02:41.688707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46561
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:02:41.693951 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.685421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.98.62:41127)
0927 15:02:41.685607 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.685611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.685658 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:02:41.685844 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.685847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.685897 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.686016 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.686504 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.687170 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.688305 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.688662 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.688771 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.690699 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.690703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.690705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.690731 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.692211 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.692660 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.692665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.692670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.692753 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.693033 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.693038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.693040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.693228 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.693333 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.693605 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.693715 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:02:41.693959 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.685291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33324)
0927 15:02:41.685576 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.685592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.685711 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.686031 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.686034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.686242 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.686354 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.686360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.687294 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.687300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.688171 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.688185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.688798 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.688804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.688950 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.690135 (+ 1185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.690155 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.690547 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.692343 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.692347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.692349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.692552 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.692853 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.692856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.692858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.692943 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.693334 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.693338 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.693437 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.693540 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.693708 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260927 15:02:41.694480 6522 tablet_copy_service.cc:140] P daf76d75dba64707a8ccd1ef2fa9a0c8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f2e6f6795b0b4b968e95ca789b89be9a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33324)
I20260927 15:02:41.694661 6522 tablet_copy_service.cc:161] P daf76d75dba64707a8ccd1ef2fa9a0c8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f2e6f6795b0b4b968e95ca789b89be9a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33324: session id = f2e6f6795b0b4b968e95ca789b89be9a-00000000000000000000000000000000
I20260927 15:02:41.696118 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:02:41.697589 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:02:41.699591 6966 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: tablet copy: Starting download of 0 data blocks...
I20260927 15:02:41.699800 6966 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: tablet copy: Starting download of 1 WAL segments...
I20260927 15:02:41.701399 6966 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:02:41.703018 6522 tablet_copy_service.cc:342] P daf76d75dba64707a8ccd1ef2fa9a0c8: Request end of tablet copy session f2e6f6795b0b4b968e95ca789b89be9a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33324
I20260927 15:02:41.703151 6522 tablet_copy_service.cc:433] P daf76d75dba64707a8ccd1ef2fa9a0c8: ending tablet copy session f2e6f6795b0b4b968e95ca789b89be9a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f2e6f6795b0b4b968e95ca789b89be9a
I20260927 15:02:41.710326 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:41.712033 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.712064 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.712304 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:41.713958 6966 server_base.cc:1034] running on GCE node
I20260927 15:02:41.714123 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:41.714424 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:41.715595 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790521361715618 us; error 57 us; skew 500 ppm
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:41.717067 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 15:02:41.717518 6966 webserver.cc:460] Webserver started at http://127.0.98.61:34767/ using document root <none> and password file <none>
I20260927 15:02:41.717706 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:41.717761 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:41.718835 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:41.719734 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.719973 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:41.720057 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "ca74d018d85e4bc2a598bca204d6dd0c"
server_key_iv: "cea60d42615b548b119872ea7cfe0813"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.720137 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:41.726861 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:41.728909 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:41.733711 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35525
I20260927 15:02:41.734228 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
I20260927 15:02:41.735690 7138 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:02:41.736608 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Bootstrap starting.
I20260927 15:02:41.742055 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35525 every 8 connection(s)
I20260927 15:02:41.744468 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:02:41.744765 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: Bootstrap complete.
I20260927 15:02:41.745229 7138 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } }
I20260927 15:02:41.745376 7138 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f2e6f6795b0b4b968e95ca789b89be9a, State: Initialized, Role: LEARNER
I20260927 15:02:41.745512 7138 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } }
I20260927 15:02:41.745836 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } }
I20260927 15:02:41.746021 7138 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:41.745993 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: LEARNER
I20260927 15:02:41.748921 7152 catalog_manager.cc:1259] Loaded cluster ID: 4911026bf1eb4171bc627eb4ce0f4c46
I20260927 15:02:41.748999 7152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: loading cluster ID for follower catalog manager: success
I20260927 15:02:41.750531 7152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: acquiring CA information for follower catalog manager: success
I20260927 15:02:41.751578 7152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:02:41.834806 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:41.835090 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:41.835165 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:41.839488 7074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:41.839576 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:41.839612 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:41.839679 7074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:41.839730 7074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:41.845118 7074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:41.846478 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:41.847710 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:41.856302 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:41.856578 7074 server_base.cc:1034] running on GCE node
W20260927 15:02:41.856304 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:41.856335 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:41.857400 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:41.858964 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:41.860262 7074 hybrid_clock.cc:648] HybridClock initialized: now 1790521361860228 us; error 51 us; skew 500 ppm
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:41.863193 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 15:02:41.864123 7074 webserver.cc:460] Webserver started at http://127.0.98.60:42993/ using document root <none> and password file <none>
I20260927 15:02:41.864702 7074 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:41.864791 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:41.865006 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:41.867106 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/instance:
uuid: "fbb0b749831846ac8eeac62a393c7eac"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "56efc6a8e946f95bd0ee03cddfddfcd9"
server_key_iv: "6ba006d089831ae15448313c4c76d734"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.867676 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal/instance:
uuid: "fbb0b749831846ac8eeac62a393c7eac"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "56efc6a8e946f95bd0ee03cddfddfcd9"
server_key_iv: "6ba006d089831ae15448313c4c76d734"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.871528 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 15:02:41.873930 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:41.875154 7074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:41.875326 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
uuid: "fbb0b749831846ac8eeac62a393c7eac"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "56efc6a8e946f95bd0ee03cddfddfcd9"
server_key_iv: "6ba006d089831ae15448313c4c76d734"
server_key_version: "encryptionkey@0"
I20260927 15:02:41.875437 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:41.898438 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:41.901877 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:41.902084 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:41.905818 6693 heartbeater.cc:503] Master 127.0.98.62:41127 was elected leader, sending a full tablet report...
I20260927 15:02:41.910409 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46561
I20260927 15:02:41.910697 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46561 every 8 connection(s)
I20260927 15:02:41.911514 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
I20260927 15:02:41.914248 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:41.917817 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:02:41.918970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 7074
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 15:02:41.931671 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.919871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.0.98.62:41127)
0927 15:02:41.920019 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.920024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.920969 (+ 945us) server_negotiation.cc:407] Connection header received
0927 15:02:41.921950 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.921953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.922002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.922130 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.922945 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.923597 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.924802 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.925039 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.925151 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.927912 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.927916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.927919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.927947 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.929526 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.930150 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.930155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.930157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.930246 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.930590 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.930593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.930594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.930780 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.930886 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.931373 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.931497 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:02:41.932471 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.920264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33334)
0927 15:02:41.920816 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.920862 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.921655 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.922179 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.922184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.922544 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.922794 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.922807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.923732 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.923735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.924675 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.924684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.925191 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.925195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.925380 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.926022 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.926047 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.927758 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.929636 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.929646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.929657 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.930048 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.930343 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.930347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.930349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.930488 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.930884 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.930892 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.931141 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.931319 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.931631 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":139,"threads_started":1}
I20260927 15:02:41.932473 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.920265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.0.98.60:46561)
0927 15:02:41.920729 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.920748 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.920786 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:02:41.921655 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.921679 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.921948 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.922132 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.922936 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.924068 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.925010 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.925379 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.925501 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.928301 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.928306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.928315 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.928336 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.930189 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.930636 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.930641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.930643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.930736 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.931021 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.931026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.931027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.931248 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.931463 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.931672 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.932077 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"spinlock_wait_cycles":74112,"thread_start_us":154,"threads_started":1}
I20260927 15:02:41.933933 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 15:02:41.942242 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.934553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.98.61:35525)
0927 15:02:41.934869 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.934872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.934901 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:02:41.934985 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.934987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.935026 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.935092 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.935451 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.936093 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.936882 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.937099 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.937247 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.939191 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.939194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.939196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.939218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.940621 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.941009 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.941012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.941013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.941082 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.941385 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.941387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.941388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.941555 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.941628 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.941998 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.942096 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":132,"threads_started":1}
I20260927 15:02:41.942314 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.934410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.0.1:33732)
0927 15:02:41.934847 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.934860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.934916 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.935120 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.935122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.935261 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.935351 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.935356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.936210 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.936213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.936788 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.936796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.937273 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.937279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.937364 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.937859 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.937872 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.939067 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.940726 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.940729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.940730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.940926 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.941158 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.941160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.941161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.941232 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.941646 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.941648 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.941809 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.941936 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.942104 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362}
I20260927 15:02:41.943154 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521361, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 15:02:41.950315 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.943627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.0.1:37314)
0927 15:02:41.943726 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.943753 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.943814 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.944024 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.944027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.944150 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.944216 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.944221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.944870 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.944872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.945378 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.945385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.945882 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.945885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.945941 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.946415 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.946427 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.947676 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.949083 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.949085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.949086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.949264 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.949489 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.949491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.949492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.949562 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.949929 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.949931 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.950002 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.950087 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.950173 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:02:41.950340 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.943763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.0.98.60:46561)
0927 15:02:41.943869 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.943872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.943882 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:02:41.943908 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.943910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.943940 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.944002 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.944307 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.944775 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.945477 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.945709 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.945867 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.947830 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.947833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.947835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.947853 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.948986 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.949345 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.949348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.949349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.949412 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.949634 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.949636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.949637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.949848 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.949923 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.950136 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.950226 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:02:41.953435 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Bootstrap starting.
I20260927 15:02:41.955766 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:41.956504 7220 log.cc:824] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:41.958508 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: No bootstrap required, opened a new log
I20260927 15:02:41.961426 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } }
I20260927 15:02:41.961637 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:41.961720 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb0b749831846ac8eeac62a393c7eac, State: Initialized, Role: FOLLOWER
I20260927 15:02:41.962218 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } }
I20260927 15:02:41.962725 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } } }
I20260927 15:02:41.962867 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:02:41.963418 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:02:41.968259 7237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:02:41.968340 7237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:02:41.971333 7074 master_runner.cc:428] Detected that this master fbb0b749831846ac8eeac62a393c7eac is joining an existing cluster
I20260927 15:02:41.971407 7074 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46561
I20260927 15:02:41.974596 7226 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:41.974773 7226 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } }
I20260927 15:02:41.975708 7226 leader_election.cc:288] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127), f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61:35525)
I20260927 15:02:41.976490 6512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb0b749831846ac8eeac62a393c7eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" is_pre_election: true
I20260927 15:02:41.976490 7106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb0b749831846ac8eeac62a393c7eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" is_pre_election: true
I20260927 15:02:41.976667 7106 raft_consensus.cc:1799] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 LEARNER]: Handling vote request from an unknown peer fbb0b749831846ac8eeac62a393c7eac
I20260927 15:02:41.976818 7106 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate fbb0b749831846ac8eeac62a393c7eac for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 15:02:41.977272 7171 leader_election.cc:302] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb0b749831846ac8eeac62a393c7eac; no voters: daf76d75dba64707a8ccd1ef2fa9a0c8, f2e6f6795b0b4b968e95ca789b89be9a
I20260927 15:02:41.977684 7226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:41.979045 7226 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 15:02:41.984102 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.974012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.0.98.61:35525)
0927 15:02:41.974129 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.974133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.974510 (+ 377us) server_negotiation.cc:407] Connection header received
0927 15:02:41.974672 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.974676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.974726 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.974780 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.975447 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.976183 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.978303 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.978643 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.978723 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.980400 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.980404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.980407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.980432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.982237 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.982699 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.982704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.982708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.982826 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.983160 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.983175 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.983177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.983396 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.983494 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.983828 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.983947 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:02:41.984118 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.974134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.0.1:33744)
0927 15:02:41.974435 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.974448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.974567 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.974919 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.974921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.975183 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.975316 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.975322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.977285 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.977291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.978157 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.978167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.978898 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.978904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.979064 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.979685 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.979706 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.980291 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.982348 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.982351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.982353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.982604 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.982944 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.982948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.982951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.983054 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.983512 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.983515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.983623 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.983755 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.983892 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":9,"thread_start_us":71,"threads_started":1}
I20260927 15:02:41.985837 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.973510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33338)
0927 15:02:41.973816 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.973833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.973967 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.974354 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.974357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.975452 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.975557 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.975563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.978958 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.978961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.979849 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.979859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.980554 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.980557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.980668 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.981602 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.981614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.982022 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.984009 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.984013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.984015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.984406 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.984685 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.984688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.984690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.984792 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.985296 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.985299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.985389 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.985483 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.985620 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260927 15:02:41.985855 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.973481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.0.98.62:41127)
0927 15:02:41.973614 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.973621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.973896 (+ 275us) server_negotiation.cc:407] Connection header received
0927 15:02:41.974086 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.974091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.974141 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.974200 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.976837 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.977704 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.979984 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.980318 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.980393 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.982165 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.982170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.982174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.982204 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.983901 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.984490 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.984494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.984496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.984581 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.984915 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.984918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.984920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.985136 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.985233 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.985590 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.985699 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:02:41.986332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6564
I20260927 15:02:41.996232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38235
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41127
--webserver_port=42817
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:02:41 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521360, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 15:02:41.998561 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.973921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.0.98.60:46561)
0927 15:02:41.974017 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.974021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.974341 (+ 320us) server_negotiation.cc:407] Connection header received
0927 15:02:41.974480 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.974483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.974516 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.974559 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.975736 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.976494 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.980870 (+ 4376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.981203 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.981269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.994802 (+ 13533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.994805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.994808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.994828 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.996730 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.997163 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.997167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.997169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.997290 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.997602 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.997607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.997611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:41.997862 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:41.997960 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:41.998271 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 15:02:41.998431 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":768}
I20260927 15:02:41.998574 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.973519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.0.1:37316)
0927 15:02:41.974264 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.974276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.974391 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.974670 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.974672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.975089 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.975230 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.975239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.977951 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.977954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.978823 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.978832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.981390 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.981393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.981489 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.994234 (+ 12745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.994248 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.994669 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.996838 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.996840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.996843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.997075 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.997404 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.997410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.997415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.997502 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:41.997996 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:41.997999 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:41.998099 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 15:02:41.998215 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 15:02:41.998367 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":109,"threads_started":1}
I20260927 15:02:42.001505 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.990798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.0.1:37332)
0927 15:02:41.990901 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:41.990915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:41.991164 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:41.991528 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:41.991534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:41.991737 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:41.991865 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.991875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.992940 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.992946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.994282 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:41.994296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.994984 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.994990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:41.995114 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.995503 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:41.995514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:41.996972 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:41.999557 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:41.999560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:41.999562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:41.999801 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.000642 (+ 841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.000645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.000646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.000732 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.001119 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.001121 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.001203 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.001285 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.001390 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 15:02:42.001627 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:41.990889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.0.98.60:46561)
0927 15:02:41.991263 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:41.991274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:41.991286 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:41.991345 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:41.991348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:41.991395 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:41.991480 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:41.991990 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.992805 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.994424 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:41.994852 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:41.995330 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:41.997134 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:41.997141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:41.997145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:41.997169 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:41.999418 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:41.999893 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:41.999897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:41.999898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.000455 (+ 557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.000830 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.000833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.000835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.001024 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.001117 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.001356 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.001463 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":94,"threads_started":1}
I20260927 15:02:42.002583 6501 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46561: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46561
}
attrs {
promote: true
}
}
dest_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8"
cas_config_opid_index: 5
I20260927 15:02:42.003290 6501 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } }
I20260927 15:02:42.005023 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fbb0b749831846ac8eeac62a393c7eac (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.006458 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46561
I20260927 15:02:42.006628 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.006721 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:42.007596 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Refusing update from remote peer daf76d75dba64707a8ccd1ef2fa9a0c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:02:42.008222 7074 master.cc:537] Master@127.0.98.60:46561 shutting down...
I20260927 15:02:42.010244 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.004553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.0.98.61:35525)
0927 15:02:42.004658 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.004662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.004675 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:02:42.004723 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.004726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.004785 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.004886 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:42.005215 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.006134 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.007995 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.008821 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.008947 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.009147 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.009984 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.010082 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:02:42.010243 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:42.010414 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.004336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.0.1:33760)
0927 15:02:42.004474 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.004491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.004574 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.004924 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.004929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.004952 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:42.005068 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.005077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.006283 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.006289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.007831 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.007845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.009219 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.009226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.009355 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.009423 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.009507 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.009616 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:02:42.011423 7106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 LEARNER]: Refusing update from remote peer daf76d75dba64707a8ccd1ef2fa9a0c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260927 15:02:42.012990 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 -> Peer fbb0b749831846ac8eeac62a393c7eac (127.0.98.60:46561): Couldn't send request to peer fbb0b749831846ac8eeac62a393c7eac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:02:42.013159 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 15:02:42.015131 7106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fbb0b749831846ac8eeac62a393c7eac (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.017148 7074 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:42.017427 7074 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:42.017513 7074 tablet_replica.cc:330] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: stopping tablet replica
I20260927 15:02:42.017478 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader daf76d75dba64707a8ccd1ef2fa9a0c8. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.017593 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: LEARNER
I20260927 15:02:42.017681 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.017758 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: LEARNER
I20260927 15:02:42.022554 7074 master.cc:552] Master@127.0.98.60:46561 shutdown complete.
I20260927 15:02:42.022655 7074 master_runner.cc:305] Clearing existing system tablet
I20260927 15:02:42.022897 7074 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:42.022946 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:42.023100 7062 raft_consensus.cc:1050] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: attempting to promote NON_VOTER f2e6f6795b0b4b968e95ca789b89be9a to VOTER
I20260927 15:02:42.023579 7062 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } }
I20260927 15:02:42.024873 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.024594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.0.1:37346)
0927 15:02:42.024724 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.024796 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:02:42.025261 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 LEARNER]: Refusing update from remote peer daf76d75dba64707a8ccd1ef2fa9a0c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 15:02:42.025715 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 15:02:42.027325 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.027062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.0.1:37352)
0927 15:02:42.027209 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.027261 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:02:42.027484 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 -> Peer fbb0b749831846ac8eeac62a393c7eac (127.0.98.60:46561): Couldn't send request to peer fbb0b749831846ac8eeac62a393c7eac. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:02:42.028151 7251 raft_consensus.cc:2932] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.028589 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.029695 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.029814 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:42.029929 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.029987 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:42.030205 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.030318 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:02:42.034117 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.002s sys 0.000s
I20260927 15:02:42.035096 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.035317 7074 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:42.035442 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:42.035475 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
uuid: "fbb0b749831846ac8eeac62a393c7eac"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "56efc6a8e946f95bd0ee03cddfddfcd9"
server_key_iv: "6ba006d089831ae15448313c4c76d734"
server_key_version: "encryptionkey@0"
I20260927 15:02:42.037536 7074 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:42.039887 7074 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:02:42.039979 7074 log.cc:1196] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 15:02:42.040217 7074 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Deleting consensus metadata
I20260927 15:02:42.040397 7074 master_runner.cc:315] Copying system tablet from 127.0.98.62:41127
I20260927 15:02:42.041530 7074 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41127
I20260927 15:02:42.054282 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.042136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.0.98.62:41127)
0927 15:02:42.042253 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.042258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.046069 (+ 3811us) server_negotiation.cc:407] Connection header received
0927 15:02:42.046245 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.046249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.046301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.046405 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.046925 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.047734 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.048962 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.049283 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.049403 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.050719 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.050722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.050725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.050751 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.052470 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.052967 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.052971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.052973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.053066 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.053375 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.053378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.053380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.053593 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.053693 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.053999 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.054113 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:02:42.054280 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.042047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33344)
0927 15:02:42.045957 (+ 3910us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.045973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.046091 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.046447 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.046450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.046677 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.046799 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.046806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.047857 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.047861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.048835 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.048845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.049404 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.049408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.049504 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.050072 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.050087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.050516 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.052605 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.052608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.052610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.052856 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.053174 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.053177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.053179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.053273 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.053694 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.053696 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.053823 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.053935 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.054076 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3835,"thread_start_us":90,"threads_started":1}
I20260927 15:02:42.055119 6522 tablet_copy_service.cc:140] P daf76d75dba64707a8ccd1ef2fa9a0c8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fbb0b749831846ac8eeac62a393c7eac ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33344)
I20260927 15:02:42.055224 6522 tablet_copy_service.cc:161] P daf76d75dba64707a8ccd1ef2fa9a0c8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fbb0b749831846ac8eeac62a393c7eac at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33344: session id = fbb0b749831846ac8eeac62a393c7eac-00000000000000000000000000000000
I20260927 15:02:42.056300 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:02:42.057636 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:02:42.060050 7074 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: tablet copy: Starting download of 0 data blocks...
I20260927 15:02:42.060333 7074 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: tablet copy: Starting download of 1 WAL segments...
I20260927 15:02:42.061950 7074 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:02:42.063737 6522 tablet_copy_service.cc:342] P daf76d75dba64707a8ccd1ef2fa9a0c8: Request end of tablet copy session fbb0b749831846ac8eeac62a393c7eac-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33344
I20260927 15:02:42.063819 6522 tablet_copy_service.cc:433] P daf76d75dba64707a8ccd1ef2fa9a0c8: ending tablet copy session fbb0b749831846ac8eeac62a393c7eac-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fbb0b749831846ac8eeac62a393c7eac
I20260927 15:02:42.071481 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:42.073283 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:42.073460 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:42.074892 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:42.074960 7074 server_base.cc:1034] running on GCE node
I20260927 15:02:42.075232 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:42.075516 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:42.076628 7074 hybrid_clock.cc:648] HybridClock initialized: now 1790521362076616 us; error 29 us; skew 500 ppm
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:42.078361 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 15:02:42.078830 7074 webserver.cc:460] Webserver started at http://127.0.98.60:44123/ using document root <none> and password file <none>
I20260927 15:02:42.079008 7074 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:42.079051 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:42.080382 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:42.081395 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.081763 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 15:02:42.081866 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
uuid: "fbb0b749831846ac8eeac62a393c7eac"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "56efc6a8e946f95bd0ee03cddfddfcd9"
server_key_iv: "6ba006d089831ae15448313c4c76d734"
server_key_version: "encryptionkey@0"
I20260927 15:02:42.081938 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:42.104831 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:42.107376 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:42.112290 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46561
I20260927 15:02:42.112892 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
I20260927 15:02:42.114514 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46561 every 8 connection(s)
I20260927 15:02:42.117050 7322 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:02:42.118150 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Bootstrap starting.
I20260927 15:02:42.129478 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:02:42.129820 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Bootstrap complete.
I20260927 15:02:42.130455 7322 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } }
I20260927 15:02:42.130605 7322 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fbb0b749831846ac8eeac62a393c7eac, State: Initialized, Role: LEARNER
I20260927 15:02:42.130721 7322 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } }
I20260927 15:02:42.131100 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.131264 7322 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:42.131283 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: This master's current role is: LEARNER
I20260927 15:02:42.134722 6827 heartbeater.cc:503] Master 127.0.98.62:41127 was elected leader, sending a full tablet report...
I20260927 15:02:42.135385 7339 catalog_manager.cc:1259] Loaded cluster ID: 4911026bf1eb4171bc627eb4ce0f4c46
I20260927 15:02:42.135478 7339 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: loading cluster ID for follower catalog manager: success
I20260927 15:02:42.137670 7339 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: acquiring CA information for follower catalog manager: success
W20260927 15:02:42.137928 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:42.138255 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:42.138335 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:02:42.139099 7339 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:02:42.143143 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:42.143198 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:42.143294 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:42.148995 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42817
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:42.150127 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:42.151154 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:42.161002 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:42.160998 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:42.160981 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:42.161480 7247 server_base.cc:1034] running on GCE node
I20260927 15:02:42.161971 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:42.162678 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:42.163964 7247 hybrid_clock.cc:648] HybridClock initialized: now 1790521362163949 us; error 35 us; skew 500 ppm
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:42.166976 7247 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 15:02:42.167902 7247 webserver.cc:460] Webserver started at http://127.0.98.1:42817/ using document root <none> and password file <none>
I20260927 15:02:42.168501 7247 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:42.168609 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:42.172981 7247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:42.175244 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.176546 7247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:42.176684 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2"
format_stamp: "Formatted at 2026-09-27 15:02:40 on dist-test-slave-ml7t"
server_key: "ec7708502d0a4d3104c6a5544459e543"
server_key_iv: "afefd4c7dc25b7d985dd1d90960e6061"
server_key_version: "encryptionkey@0"
I20260927 15:02:42.177248 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:42.197010 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:42.199983 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:42.200186 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:42.200784 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:42.201663 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:42.201738 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.201821 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:42.201867 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.211218 7247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38235
I20260927 15:02:42.211234 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38235 every 8 connection(s)
I20260927 15:02:42.212186 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:42.218986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 7247
I20260927 15:02:42.219167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6698
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:42.227560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34969
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41127
--webserver_port=43405
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:02:42.229301 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.214054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38431 (local address 127.0.98.60:46561)
0927 15:02:42.214321 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.214324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.215248 (+ 924us) server_negotiation.cc:407] Connection header received
0927 15:02:42.216436 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.216439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.216493 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.216552 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.217865 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.218925 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220345 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.220690 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220755 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.224685 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.224689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.224692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.224719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.226837 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.227472 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.227476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.227478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.227565 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.227968 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.227974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.227978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.228226 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.228300 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.228967 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.229104 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260927 15:02:42.229444 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.214063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42509 (local address 127.0.98.61:35525)
0927 15:02:42.214205 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.214212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.215517 (+ 1305us) server_negotiation.cc:407] Connection header received
0927 15:02:42.216436 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.216439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.216495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.216550 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.217866 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.218642 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220334 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.220678 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220755 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.225082 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.225089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.225095 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.225132 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.227676 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.228184 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.228188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.228189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.228271 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.228570 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.228573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.228575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.228778 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.228852 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.229185 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.229301 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:02:42.229724 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.215853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37293 (local address 127.0.98.62:41127)
0927 15:02:42.215978 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.215983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.216040 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:02:42.216408 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.216412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.216464 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.216544 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.217850 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.218381 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220327 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.220749 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220833 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.225120 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.225126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.225132 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.225163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.227410 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.228247 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.228254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.228259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.228354 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.228671 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.228677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.228682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.228931 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.229011 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.229386 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.229515 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:02:42.230919 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.214529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.98.1:42509)
0927 15:02:42.215429 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.215442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.216199 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.216725 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.216735 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.217227 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.217685 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.217691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.218783 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.218786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.220197 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.220208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.220942 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.221268 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.222315 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.222332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.224930 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.227780 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.227783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.227785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.228081 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.228390 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.228393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.228394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.228473 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.228924 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.228927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.229011 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.229661 (+ 650us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.229853 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":392,"spinlock_wait_cycles":71040,"thread_start_us":171,"threads_started":1}
I20260927 15:02:42.231235 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.214529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.98.1:38431)
0927 15:02:42.215076 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.215116 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.216199 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.216725 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.216735 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.217229 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.217683 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.217691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.219113 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.219116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.220230 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.220238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.220942 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.220946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.221432 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.222053 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.222076 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.224476 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.226962 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.226971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.226986 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.227345 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.227676 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.227679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.227681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.227856 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.228397 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.228403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.228702 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.229447 (+ 745us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.229810 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":445,"spinlock_wait_cycles":12544,"thread_start_us":651,"threads_started":1}
I20260927 15:02:42.230952 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.215709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:37293)
0927 15:02:42.215965 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.215978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.216221 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.216739 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.216741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.217196 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.217676 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.217697 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.218517 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.218520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.220183 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.220196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.221007 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.221013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.221265 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.222389 (+ 1124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.222409 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.224965 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.227530 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.227533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.227535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.228130 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.228469 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.228475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.228480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.228571 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.229096 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.229099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.229197 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.230469 (+ 1272us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.230654 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":104,"thread_start_us":103,"threads_started":1}
I20260927 15:02:42.232251 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46561
I20260927 15:02:42.232496 7465 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.232944 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260927 15:02:42.233024 7466 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.233069 7465 heartbeater.cc:511] Master 127.0.98.60:46561 requested a full tablet report, sending...
I20260927 15:02:42.233157 7466 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260927 15:02:42.233587 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35525
I20260927 15:02:42.233655 7467 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.233850 7467 heartbeater.cc:511] Master 127.0.98.61:35525 requested a full tablet report, sending...
I20260927 15:02:42.234772 7102 ts_manager.cc:194] Registered new tserver with Master: e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1:38235)
I20260927 15:02:42.234800 7286 ts_manager.cc:194] Registered new tserver with Master: e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1:38235)
I20260927 15:02:42.235551 6501 ts_manager.cc:194] Re-registered known tserver with Master: e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1:38235)
I20260927 15:02:42.236433 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37293
W20260927 15:02:42.340310 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:42.340574 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:42.340616 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:42.345163 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:42.345233 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:42.345309 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:42.351632 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43405
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:42.352869 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:42.353948 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:42.363883 7479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:42.363914 7473 server_base.cc:1034] running on GCE node
W20260927 15:02:42.363848 7478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:42.363873 7481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:42.364494 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:42.365096 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:42.366274 7473 hybrid_clock.cc:648] HybridClock initialized: now 1790521362366244 us; error 43 us; skew 500 ppm
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:42.368891 7473 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 15:02:42.369859 7473 webserver.cc:460] Webserver started at http://127.0.98.2:43405/ using document root <none> and password file <none>
I20260927 15:02:42.370425 7473 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:42.370672 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:42.374984 7473 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:42.376518 6961 heartbeater.cc:503] Master 127.0.98.62:41127 was elected leader, sending a full tablet report...
I20260927 15:02:42.377355 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.378870 7473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:42.378998 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "ee668e13116e4035b8861fd5af19cfa3"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "6ecba537353ea46d53b7f04b593e2551"
server_key_iv: "6d7a4419d9ecc8d350ece2e71aacc676"
server_key_version: "encryptionkey@0"
I20260927 15:02:42.379648 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:42.405352 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:42.408668 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:42.408871 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:42.409461 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:42.410374 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:42.410452 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.410522 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:42.410573 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.419099 7473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34969
I20260927 15:02:42.419160 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34969 every 8 connection(s)
I20260927 15:02:42.420282 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:42.424722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 7473
I20260927 15:02:42.424928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6832
I20260927 15:02:42.431898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46395
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41127
--webserver_port=41833
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 15:02:42.440701 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.422365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35261 (local address 127.0.98.60:46561)
0927 15:02:42.422498 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.422502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.423626 (+ 1124us) server_negotiation.cc:407] Connection header received
0927 15:02:42.425002 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.425008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.425079 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.425168 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.427304 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.428148 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.430660 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.431041 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.431129 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.436173 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.436181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.436186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.436224 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.438034 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.438700 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.438703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.438705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.438794 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.439222 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.439225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.439227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.439469 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.439560 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.440394 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.440531 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:02:42.441597 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.422823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:36345)
0927 15:02:42.423780 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.423795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.424730 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.426266 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.426269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.426447 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.427112 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.427131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.428337 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.428340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.430516 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.430522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.431427 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.431431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.431671 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.432639 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.432667 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.436060 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.439333 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.439338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.439342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.439639 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.440028 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.440033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.440037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.440146 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.440759 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.440766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.440874 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.440987 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.441166 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":447,"spinlock_wait_cycles":88832,"thread_start_us":188,"threads_started":1}
I20260927 15:02:42.441998 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.424112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.98.2:58625)
0927 15:02:42.424459 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.424487 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.424739 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.426753 (+ 2014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.426758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.426975 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.427154 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.427165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.428350 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.428356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.430521 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.430534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.431484 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.431488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.431637 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.433078 (+ 1441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.433116 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.435531 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.438891 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.438898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.438903 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.439211 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.439500 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.439506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.439511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.439603 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.440041 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.440044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.440149 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.440236 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.440548 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":480,"spinlock_wait_cycles":58496,"thread_start_us":80,"threads_started":1}
I20260927 15:02:42.442534 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.424247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58625 (local address 127.0.98.61:35525)
0927 15:02:42.424365 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.424369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.424636 (+ 267us) server_negotiation.cc:407] Connection header received
0927 15:02:42.426451 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.426458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.426522 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.426585 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.427296 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.428204 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.430712 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.431179 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.431269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.435789 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.435793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.435796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.435827 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.438717 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.439310 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.439313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.439314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.439392 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.439700 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.439703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.439705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.439909 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.439998 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.442327 (+ 2329us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.442428 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:02:42.442998 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.422823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.98.2:35261)
0927 15:02:42.423415 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.423482 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.424730 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.425638 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.425651 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.426227 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.427112 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.427131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.428337 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.428340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.430488 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.430502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.431350 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.431354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.431623 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.433942 (+ 2319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.433955 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.436030 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.438185 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.438193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.438214 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.438560 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.438900 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.438904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.438910 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.439101 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.439608 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.439614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.439944 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.440158 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.442817 (+ 2659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":3170,"spinlock_wait_cycles":1217792,"thread_start_us":703,"threads_started":1}
I20260927 15:02:42.443629 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35525
I20260927 15:02:42.443636 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46561
I20260927 15:02:42.443864 7602 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.443866 7604 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.444111 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.422371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36345 (local address 127.0.98.62:41127)
0927 15:02:42.422519 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.422526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.423880 (+ 1354us) server_negotiation.cc:407] Connection header received
0927 15:02:42.425074 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.425078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.425137 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.425233 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.427303 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.428147 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.430668 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.431038 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.431140 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.436255 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.436259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.436262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.436290 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.438729 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.439781 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.439788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.439793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.439905 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.440275 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.440281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.440286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.440587 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.440704 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.442378 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.442554 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:02:42.444515 7602 heartbeater.cc:511] Master 127.0.98.60:46561 requested a full tablet report, sending...
I20260927 15:02:42.444743 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260927 15:02:42.444830 7603 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.445044 7603 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260927 15:02:42.445107 7604 heartbeater.cc:511] Master 127.0.98.61:35525 requested a full tablet report, sending...
I20260927 15:02:42.445680 7286 ts_manager.cc:194] Registered new tserver with Master: ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969)
I20260927 15:02:42.445842 6501 ts_manager.cc:194] Re-registered known tserver with Master: ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969)
I20260927 15:02:42.446899 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36345
I20260927 15:02:42.447665 7102 ts_manager.cc:194] Registered new tserver with Master: ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969)
W20260927 15:02:42.557889 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:42.558156 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:42.558230 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:42.562304 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:42.562366 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:42.562455 7609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:42.567796 7609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41833
--tserver_master_addrs=127.0.98.62:41127,127.0.98.61:35525,127.0.98.60:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:42.568898 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:42.570077 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:42.579737 7615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:42.579737 7616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:42.579867 7618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:42.579968 7609 server_base.cc:1034] running on GCE node
I20260927 15:02:42.580451 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:42.581108 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:42.582321 7609 hybrid_clock.cc:648] HybridClock initialized: now 1790521362582290 us; error 48 us; skew 500 ppm
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:42.585170 7609 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 15:02:42.586061 7609 webserver.cc:460] Webserver started at http://127.0.98.3:41833/ using document root <none> and password file <none>
I20260927 15:02:42.586630 7609 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:42.586728 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:42.590657 7609 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:42.592723 7625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.593972 7609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:42.594087 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "ef32965fdc614ddb9312d28c310ba524"
format_stamp: "Formatted at 2026-09-27 15:02:41 on dist-test-slave-ml7t"
server_key: "4dde937574dc3d8a9559c4e93e9a76aa"
server_key_iv: "57629a19d5a3898a6320f7ebed300a0b"
server_key_version: "encryptionkey@0"
I20260927 15:02:42.594609 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:42.600304 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.596544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.0.1:37354)
0927 15:02:42.596787 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.596803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.596948 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.597257 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.597260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.597278 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:42.597386 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.597391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.598181 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.598185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.598961 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.598969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.599874 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.599878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.599973 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.599995 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.600083 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.600170 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260927 15:02:42.600301 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.596617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.0.98.60:46561)
0927 15:02:42.596777 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.596782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.596890 (+ 108us) server_negotiation.cc:407] Connection header received
0927 15:02:42.597062 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.597065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.597119 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.597232 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:42.597493 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.598069 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.599086 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.599713 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.599859 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.600000 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.600114 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.600179 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:02:42.600907 7296 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 15:02:42.601264 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader daf76d75dba64707a8ccd1ef2fa9a0c8. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.601397 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: This master's current role is: LEARNER
I20260927 15:02:42.615418 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:42.618672 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:42.618875 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:42.619484 7609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:42.620434 7609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:42.620504 7609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.620570 7609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:42.620625 7609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:42.630111 7609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46395
I20260927 15:02:42.630134 7739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46395 every 8 connection(s)
I20260927 15:02:42.631209 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:42.634402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 7609
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 15:02:42 dist-test-slave-ml7t krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521362, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:42.648168 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.636772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.98.62:41127)
0927 15:02:42.637076 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.637081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.637095 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:42.637133 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.637136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.637189 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.637272 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.638365 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.639158 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640292 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.640639 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640743 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.643362 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.643369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.643376 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.643406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.645544 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.646077 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.646080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.646082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.646166 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.646523 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.646529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.646574 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.646853 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.646991 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.647822 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.647951 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260927 15:02:42.649842 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.633002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34607 (local address 127.0.98.60:46561)
0927 15:02:42.633186 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.633192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.634915 (+ 1723us) server_negotiation.cc:407] Connection header received
0927 15:02:42.635350 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.635356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.635431 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.635521 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.637191 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.637998 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640039 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.640378 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640445 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.644068 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.644071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.644074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.644099 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.645950 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.648496 (+ 2546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.648500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.648502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.648585 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.648942 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.648945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.648947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.649131 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.649218 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.649560 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.649671 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:02:42.650437 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.633442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.98.3:41687)
0927 15:02:42.633980 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.634031 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.635096 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.637636 (+ 2540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.637639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.637863 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.637969 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.637976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.639428 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.639431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.640249 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.640257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.640899 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.640997 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.641688 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.641702 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.644508 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.647145 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.647148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.647150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.647387 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.647682 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.647684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.647686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.647880 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.648396 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.648403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.648747 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.648967 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.649300 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":678,"threads_started":1}
I20260927 15:02:42.651225 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.634056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41687 (local address 127.0.98.61:35525)
0927 15:02:42.636206 (+ 2150us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.636222 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.636236 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:42.636278 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.636281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.636327 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.636410 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.638111 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.639280 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640366 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.640694 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640788 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.644647 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.644652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.644654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.644678 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.647023 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.647496 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.647501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.647504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.647585 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.647994 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.647998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.648003 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.648238 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.648338 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.650970 (+ 2632us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.651080 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2057}
I20260927 15:02:42.651796 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.636343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.0.98.60:46561)
0927 15:02:42.637789 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.637795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.637809 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:42.637851 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.637856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.637907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.637997 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.639267 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.640029 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.641175 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.641568 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.641669 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.643932 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.643936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.643939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.643963 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.647992 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.650136 (+ 2144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.650139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.650141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.650231 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.650525 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.650530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.650533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.650728 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.650832 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.651528 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.651643 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1340,"thread_start_us":68,"threads_started":1}
I20260927 15:02:42.651291 6501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:02:42.654567 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.633442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.98.3:34607)
0927 15:02:42.634804 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.634817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.635096 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.635829 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.635838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.636342 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.637028 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.637047 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.638587 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.638591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.639912 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.639921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.640620 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.640624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.640827 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.641429 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.641448 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.643898 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.646281 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.646290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.646309 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.646680 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.648701 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.648705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.648707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.648835 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.649295 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.649298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.649383 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.650029 (+ 646us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.654389 (+ 4360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"mutex_wait_us":49,"spinlock_wait_cycles":28032,"thread_start_us":145,"threads_started":1}
I20260927 15:02:42.654724 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35525
I20260927 15:02:42.654965 7742 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.655494 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46561
I20260927 15:02:42.655534 7742 heartbeater.cc:511] Master 127.0.98.61:35525 requested a full tablet report, sending...
I20260927 15:02:42.655582 7740 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.655648 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.636659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54623 (local address 127.0.98.62:41127)
0927 15:02:42.636981 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.636986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.637000 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:42.637043 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.637045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.637095 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.637176 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.642424 (+ 5248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.643236 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.644668 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.645028 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.645113 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.647688 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.647692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.647695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.647716 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.649368 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.654368 (+ 5000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.654372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.654373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.654457 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.654777 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.654781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.654783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.654976 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.655062 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.655396 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.655507 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260927 15:02:42.655781 7740 heartbeater.cc:511] Master 127.0.98.60:46561 requested a full tablet report, sending...
I20260927 15:02:42.656066 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.634540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:54623)
0927 15:02:42.635625 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.635636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.635704 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.641200 (+ 5496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.641203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.641415 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:42.641522 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.641533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.643375 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.643378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.644168 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.644176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.645306 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.645315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.645437 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.645954 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:42.645971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:42.647545 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:42.649492 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.649495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.649496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.654248 (+ 4752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.654569 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:42.654571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:42.654573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:42.654658 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:42.655118 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:42.655121 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:42.655211 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.655773 (+ 562us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.655912 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":87,"threads_started":1}
I20260927 15:02:42.656793 7102 ts_manager.cc:194] Registered new tserver with Master: ef32965fdc614ddb9312d28c310ba524 (127.0.98.3:46395)
I20260927 15:02:42.656859 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41127
I20260927 15:02:42.656920 7741 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:42.656939 7286 ts_manager.cc:194] Registered new tserver with Master: ef32965fdc614ddb9312d28c310ba524 (127.0.98.3:46395)
I20260927 15:02:42.657073 7741 heartbeater.cc:511] Master 127.0.98.62:41127 requested a full tablet report, sending...
I20260927 15:02:42.657840 6502 ts_manager.cc:194] Re-registered known tserver with Master: ef32965fdc614ddb9312d28c310ba524 (127.0.98.3:46395)
I20260927 15:02:42.658535 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.637465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.0.98.61:35525)
0927 15:02:42.643226 (+ 5761us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.643239 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.643254 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:02:42.643291 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.643294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.643340 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.649726 (+ 6386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.649838 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.651129 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.652537 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.652764 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.652871 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.654911 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.654915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.654917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.654940 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.656730 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.657223 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.657226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.657228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.657314 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.657647 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.657650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.657652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.657919 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.658045 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.658263 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.658381 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5691,"thread_start_us":80,"threads_started":1}
I20260927 15:02:42.658573 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54623
W20260927 15:02:42.660272 6501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:42.662650 7613 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: true } } }
I20260927 15:02:42.662812 7613 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:42.665921 7760 raft_consensus.cc:1050] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: attempting to promote NON_VOTER fbb0b749831846ac8eeac62a393c7eac to VOTER
I20260927 15:02:42.666247 7760 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:42.667343 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Refusing update from remote peer daf76d75dba64707a8ccd1ef2fa9a0c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 15:02:42.667835 7760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 15:02:42.669340 7758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fbb0b749831846ac8eeac62a393c7eac (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.669729 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 LEARNER]: Refusing update from remote peer daf76d75dba64707a8ccd1ef2fa9a0c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 15:02:42.669754 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fbb0b749831846ac8eeac62a393c7eac (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.670184 7759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 15:02:42.670910 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.671023 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:42.670948 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.671079 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:42.672096 7296 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fbb0b749831846ac8eeac62a393c7eac (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.675575 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.675702 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:02:42.676021 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.676162 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:02:42.676465 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:42.676587 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:42.680953 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.675347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46395 (local address 127.0.0.1:50588)
0927 15:02:42.675551 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.675564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.675656 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.676778 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.676783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.676804 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:42.676904 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.676911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.678284 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.678288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.679495 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.679505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.680570 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.680573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.680654 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.680677 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.680741 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.680813 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":29,"thread_start_us":89,"threads_started":1}
I20260927 15:02:42.680986 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.675738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.0.98.3:46395)
0927 15:02:42.676101 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.676113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.676138 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:02:42.676220 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.676226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.676464 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.676627 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:42.677030 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.678163 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.679634 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.680460 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.680566 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.680725 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.680775 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.680863 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260927 15:02:42.682119 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.674159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38235 (local address 127.0.0.1:40460)
0927 15:02:42.674453 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.674468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.674547 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.676865 (+ 2318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.676868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.676887 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:42.676973 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.676979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.678864 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.678870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.679811 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.679821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.681654 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.681658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.681817 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.681843 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.681915 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.681985 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":28}
I20260927 15:02:42.682235 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.674719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.0.98.1:38235)
0927 15:02:42.676402 (+ 1683us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.676413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.676442 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:02:42.676508 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.676516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.676712 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.676857 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:42.677732 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.678750 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.679965 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.681549 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.681654 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.681825 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.681951 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.682085 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1592,"thread_start_us":146,"threads_started":1}
I20260927 15:02:42.682384 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.675023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34969 (local address 127.0.0.1:49032)
0927 15:02:42.675256 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.675272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.675353 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.677561 (+ 2208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.677566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.677585 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:42.677683 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.677690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.679088 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.679091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.680169 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.680179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.681647 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.681651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.681789 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.681810 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.681864 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.681937 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":6,"thread_start_us":93,"threads_started":1}
I20260927 15:02:42.682790 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.675192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.0.98.2:34969)
0927 15:02:42.677151 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.677162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.677183 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:02:42.677253 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.677258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.677439 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.677592 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:42.677843 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.678961 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.680356 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.681473 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.682061 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.682465 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.682519 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.682601 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1846,"thread_start_us":127,"threads_started":1}
I20260927 15:02:42.684474 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5 (DEFAULT_TABLE table=test-table [id=605ff00621514fa484dcf16111f73a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:42.685001 7399 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5 (DEFAULT_TABLE table=test-table [id=605ff00621514fa484dcf16111f73a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:42.685628 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:42.685370 7536 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5 (DEFAULT_TABLE table=test-table [id=605ff00621514fa484dcf16111f73a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:42.685875 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:42.686280 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:42.692696 7774 tablet_bootstrap.cc:492] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Bootstrap starting.
I20260927 15:02:42.694047 7776 tablet_bootstrap.cc:492] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Bootstrap starting.
I20260927 15:02:42.695335 7774 tablet_bootstrap.cc:654] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:42.695920 7776 tablet_bootstrap.cc:654] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:42.696568 7774 log.cc:824] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:42.696578 7776 log.cc:824] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:42.697335 7775 tablet_bootstrap.cc:492] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Bootstrap starting.
I20260927 15:02:42.698719 7774 tablet_bootstrap.cc:492] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: No bootstrap required, opened a new log
I20260927 15:02:42.698932 7774 ts_tablet_manager.cc:1397] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:02:42.699293 7775 tablet_bootstrap.cc:654] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:42.700017 7775 log.cc:824] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:42.701512 7776 tablet_bootstrap.cc:492] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: No bootstrap required, opened a new log
I20260927 15:02:42.701719 7776 ts_tablet_manager.cc:1397] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260927 15:02:42.703037 7774 raft_consensus.cc:348] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.703253 7774 raft_consensus.cc:374] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:42.703327 7774 raft_consensus.cc:729] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58a3ca2f5f74e0faed62bbb2bd313e2, State: Initialized, Role: FOLLOWER
I20260927 15:02:42.703801 7774 consensus_queue.cc:260] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.704540 7466 heartbeater.cc:503] Master 127.0.98.62:41127 was elected leader, sending a full tablet report...
I20260927 15:02:42.704856 7774 ts_tablet_manager.cc:1428] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:02:42.705171 7776 raft_consensus.cc:348] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.705417 7776 raft_consensus.cc:374] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:42.705484 7776 raft_consensus.cc:729] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef32965fdc614ddb9312d28c310ba524, State: Initialized, Role: FOLLOWER
I20260927 15:02:42.705935 7776 consensus_queue.cc:260] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.706837 7776 ts_tablet_manager.cc:1428] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:02:42.707451 7775 tablet_bootstrap.cc:492] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: No bootstrap required, opened a new log
I20260927 15:02:42.707695 7775 ts_tablet_manager.cc:1397] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 15:02:42.708715 7741 heartbeater.cc:503] Master 127.0.98.62:41127 was elected leader, sending a full tablet report...
I20260927 15:02:42.710323 7781 raft_consensus.cc:482] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:42.710461 7781 raft_consensus.cc:504] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.712742 7775 raft_consensus.cc:348] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.713059 7775 raft_consensus.cc:374] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:42.713169 7775 raft_consensus.cc:729] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee668e13116e4035b8861fd5af19cfa3, State: Initialized, Role: FOLLOWER
W20260927 15:02:42.713727 7468 tablet.cc:2369] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:42.713660 7775 consensus_queue.cc:260] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.714730 7775 ts_tablet_manager.cc:1428] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 15:02:42.714949 7781 leader_election.cc:288] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1:38235), ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969)
I20260927 15:02:42.715852 7603 heartbeater.cc:503] Master 127.0.98.62:41127 was elected leader, sending a full tablet report...
I20260927 15:02:42.717144 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.712674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38235 (local address 127.0.98.3:53821)
0927 15:02:42.712836 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.712854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.712948 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.713275 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.713279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.713296 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:42.713419 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.713427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.714476 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.714479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.715688 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.715700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.716636 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.716639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.716755 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.716787 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.716859 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.716934 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":24}
I20260927 15:02:42.717540 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.713204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34969 (local address 127.0.98.3:53097)
0927 15:02:42.713428 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:42.713442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:42.713534 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:42.713934 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:42.713937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:42.713955 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:42.714067 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.714073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.714945 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.714948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.716048 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:42.716059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.717067 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.717071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:42.717179 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.717221 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:02:42.717306 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:02:42.717388 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":4}
I20260927 15:02:42.717702 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.712760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53821 (local address 127.0.98.1:38235)
0927 15:02:42.712868 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.712875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.712895 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:02:42.713095 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.713098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.713153 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.713268 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:42.713582 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.714345 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.715870 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.716484 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.716590 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.716661 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.717488 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.717580 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:02:42.718048 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.713457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53097 (local address 127.0.98.2:34969)
0927 15:02:42.713596 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.713601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.713615 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:42.713661 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.713664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.713711 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.713827 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:42.714194 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.714825 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.716231 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.716835 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.716941 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.717005 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.717815 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.717909 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:02:42.718425 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b41aa0a9c432499ed59e8fa88a8b5" candidate_uuid: "ef32965fdc614ddb9312d28c310ba524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" is_pre_election: true
I20260927 15:02:42.718719 7419 raft_consensus.cc:2454] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef32965fdc614ddb9312d28c310ba524 in term 0.
I20260927 15:02:42.718734 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b41aa0a9c432499ed59e8fa88a8b5" candidate_uuid: "ef32965fdc614ddb9312d28c310ba524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee668e13116e4035b8861fd5af19cfa3" is_pre_election: true
I20260927 15:02:42.718967 7556 raft_consensus.cc:2454] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef32965fdc614ddb9312d28c310ba524 in term 0.
I20260927 15:02:42.719376 7628 leader_election.cc:302] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e58a3ca2f5f74e0faed62bbb2bd313e2, ef32965fdc614ddb9312d28c310ba524; no voters:
I20260927 15:02:42.719693 7781 raft_consensus.cc:2781] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:42.719798 7781 raft_consensus.cc:482] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:42.719857 7781 raft_consensus.cc:3037] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:42.721016 7781 raft_consensus.cc:504] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.721437 7781 leader_election.cc:288] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [CANDIDATE]: Term 1 election: Requested vote from peers e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1:38235), ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969)
I20260927 15:02:42.721845 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b41aa0a9c432499ed59e8fa88a8b5" candidate_uuid: "ef32965fdc614ddb9312d28c310ba524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee668e13116e4035b8861fd5af19cfa3"
I20260927 15:02:42.721865 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b41aa0a9c432499ed59e8fa88a8b5" candidate_uuid: "ef32965fdc614ddb9312d28c310ba524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2"
I20260927 15:02:42.721959 7556 raft_consensus.cc:3037] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:42.721959 7419 raft_consensus.cc:3037] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:42.723162 7556 raft_consensus.cc:2454] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef32965fdc614ddb9312d28c310ba524 in term 1.
I20260927 15:02:42.723157 7419 raft_consensus.cc:2454] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef32965fdc614ddb9312d28c310ba524 in term 1.
I20260927 15:02:42.723475 7627 leader_election.cc:302] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee668e13116e4035b8861fd5af19cfa3, ef32965fdc614ddb9312d28c310ba524; no voters:
I20260927 15:02:42.723659 7781 raft_consensus.cc:2781] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:42.723767 7781 raft_consensus.cc:686] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 1 LEADER]: Becoming Leader. State: Replica: ef32965fdc614ddb9312d28c310ba524, State: Running, Role: LEADER
I20260927 15:02:42.724064 7781 consensus_queue.cc:237] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:42.727360 6500 catalog_manager.cc:5495] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef32965fdc614ddb9312d28c310ba524 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ef32965fdc614ddb9312d28c310ba524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } health_report { overall_health: HEALTHY } } }
I20260927 15:02:42.742960 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.739229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.0.98.3:46395)
0927 15:02:42.739341 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.739344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.739356 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:42.739509 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.739512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.739566 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.739650 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:42.739905 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.740397 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.741539 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.741759 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.741895 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.742210 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:42.742587 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:42.742677 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.742747 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.742795 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:02:42.749142 7419 raft_consensus.cc:1261] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 FOLLOWER]: Refusing update from remote peer ef32965fdc614ddb9312d28c310ba524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:42.749142 7556 raft_consensus.cc:1261] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Refusing update from remote peer ef32965fdc614ddb9312d28c310ba524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:42.749929 7781 consensus_queue.cc:1035] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:42.750188 7782 consensus_queue.cc:1035] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:42.757598 7787 mvcc.cc:205] Tried to move back new op lower bound from 7333975501812744192 to 7333975501720256512. Current Snapshot: MvccSnapshot[applied={T|T < 7333975501812744192}]
I20260927 15:02:42.761793 7788 mvcc.cc:205] Tried to move back new op lower bound from 7333975501812744192 to 7333975501720256512. Current Snapshot: MvccSnapshot[applied={T|T < 7333975501812744192}]
I20260927 15:02:42.769920 6500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:42.770108 6500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:42.774299 6500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daf76d75dba64707a8ccd1ef2fa9a0c8: Sending DeleteTablet for 3 replicas of tablet 0c6b41aa0a9c432499ed59e8fa88a8b5
I20260927 15:02:42.774995 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40460
I20260927 15:02:42.775070 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49032
I20260927 15:02:42.775887 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6b41aa0a9c432499ed59e8fa88a8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50588
I20260927 15:02:42.776324 7798 tablet_replica.cc:330] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: stopping tablet replica
I20260927 15:02:42.776365 7797 tablet_replica.cc:330] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: stopping tablet replica
I20260927 15:02:42.776672 7797 raft_consensus.cc:2229] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:42.776672 7798 raft_consensus.cc:2229] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:42.776998 7797 raft_consensus.cc:2258] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:42.777197 7798 raft_consensus.cc:2258] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:42.778705 7797 ts_tablet_manager.cc:1905] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:42.779347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6470
I20260927 15:02:42.782568 7797 ts_tablet_manager.cc:1918] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:42.782684 7797 log.cc:1196] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/0c6b41aa0a9c432499ed59e8fa88a8b5
I20260927 15:02:42.783017 7797 ts_tablet_manager.cc:1939] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ee668e13116e4035b8861fd5af19cfa3: Deleting consensus metadata
I20260927 15:02:42.784457 7798 ts_tablet_manager.cc:1905] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:42.787067 7798 ts_tablet_manager.cc:1918] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:42.787194 7798 log.cc:1196] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/0c6b41aa0a9c432499ed59e8fa88a8b5
I20260927 15:02:42.787546 7798 ts_tablet_manager.cc:1939] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P ef32965fdc614ddb9312d28c310ba524: Deleting consensus metadata
W20260927 15:02:42.787588 7491 connection.cc:535] server connection from 127.0.0.1:49032 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 15:02:42.788614 7627 connection.cc:441] server connection from 127.0.0.1:50588 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50588 (request call id 1) could send its response
I20260927 15:02:42.789067 7796 tablet_replica.cc:330] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: stopping tablet replica
I20260927 15:02:42.789431 7796 raft_consensus.cc:2229] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:42.789804 7796 raft_consensus.cc:2258] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:42.791738 7796 ts_tablet_manager.cc:1905] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:42.794843 7796 ts_tablet_manager.cc:1918] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:42.794957 7796 log.cc:1196] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/0c6b41aa0a9c432499ed59e8fa88a8b5
I20260927 15:02:42.795310 7796 ts_tablet_manager.cc:1939] T 0c6b41aa0a9c432499ed59e8fa88a8b5 P e58a3ca2f5f74e0faed62bbb2bd313e2: Deleting consensus metadata
W20260927 15:02:42.796545 7354 connection.cc:441] server connection from 127.0.0.1:40460 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40460 (request call id 1) could send its response
I20260927 15:02:42.796881 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.777967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.98.61:35525)
0927 15:02:42.778076 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.778081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.786140 (+ 8059us) server_negotiation.cc:407] Connection header received
0927 15:02:42.786416 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.786419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.786474 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.786527 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.787885 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.788616 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.790047 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.790330 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.790392 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.792948 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.792952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.792955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.792984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.794791 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.795478 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.795484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.795489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.795576 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.795901 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.795905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.795906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.796109 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.796205 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.796550 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.796716 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:02:42.803298 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.792497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.0.98.61:35525)
0927 15:02:42.792607 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.792612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.793352 (+ 740us) server_negotiation.cc:407] Connection header received
0927 15:02:42.793573 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.793576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.793622 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.793673 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.794524 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.795299 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.797243 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.797591 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.797667 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.799481 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.799485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.799488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.799511 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.801184 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.802014 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.802017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.802019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.802109 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.802395 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.802398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.802400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.802592 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.802701 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.803025 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.803143 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:02:42.806667 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.797020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.0.98.60:46561)
0927 15:02:42.797138 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.797143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.797157 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:42.797215 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.797220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.797266 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.797357 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.798599 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.799453 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.800616 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.800935 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.801006 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.802727 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.802734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.802740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.802769 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.804842 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.805349 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.805353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.805355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.805434 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.805758 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.805763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.805801 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.805976 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.806055 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.806407 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.806521 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:02:42.807647 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:42.777724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.98.60:46561)
0927 15:02:42.777994 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:42.777999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:42.798660 (+ 20661us) server_negotiation.cc:407] Connection header received
0927 15:02:42.798820 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:42.798823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:42.798865 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:42.798912 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:42.799634 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.800391 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.801481 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:42.801867 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:42.801971 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:42.803919 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:42.803925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:42.803928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:42.803959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:42.805678 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.806309 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.806315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.806317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.806398 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:42.806680 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:42.806684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:42.806686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:42.806875 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:42.806995 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:42.807384 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 15:02:42.807499 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260927 15:02:43.762709 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:43.762132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:34375)
0927 15:02:43.762473 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:43.762579 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":73,"threads_started":1}
I20260927 15:02:43.762794 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:43.762131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:34733)
0927 15:02:43.762435 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:43.762692 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
W20260927 15:02:43.763185 7603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
W20260927 15:02:43.763393 7741 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:43.764333 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:43.763978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:53743)
0927 15:02:43.764194 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:43.764260 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
W20260927 15:02:43.764678 7466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41127 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:44.274839 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader daf76d75dba64707a8ccd1ef2fa9a0c8)
I20260927 15:02:44.275048 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:44.275367 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader daf76d75dba64707a8ccd1ef2fa9a0c8)
I20260927 15:02:44.275497 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:44.276003 7810 leader_election.cc:288] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127), f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61:35525)
I20260927 15:02:44.276249 7809 leader_election.cc:288] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127), fbb0b749831846ac8eeac62a393c7eac (127.0.98.60:46561)
I20260927 15:02:44.276782 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.276150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33790)
0927 15:02:44.276563 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.276667 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
I20260927 15:02:44.277007 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.276403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33806)
0927 15:02:44.276728 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.276797 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":60,"threads_started":1}
W20260927 15:02:44.277055 7273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
W20260927 15:02:44.277247 7089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:44.278013 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.277718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33818)
0927 15:02:44.277867 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.277923 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 15:02:44.278146 7273 leader_election.cc:334] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:44.278573 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.278346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33824)
0927 15:02:44.278464 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.278514 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:02:44.278755 7089 leader_election.cc:334] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:44.280791 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.276836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.0.98.60:46561)
0927 15:02:44.277221 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:44.277226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:44.277238 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:44.277294 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:44.277300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:44.277349 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:44.277787 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:44.277909 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.278675 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.279749 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.280212 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.280313 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:44.280374 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:44.280605 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 15:02:44.280677 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":316,"thread_start_us":58,"threads_started":1}
I20260927 15:02:44.280802 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.276447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46561 (local address 127.0.0.1:37400)
0927 15:02:44.277035 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.277051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:44.277162 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:44.277483 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:44.277487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:44.277525 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:44.277655 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:44.277663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.278793 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.278796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:44.279615 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:44.279623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.280312 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.280315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:44.280481 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:44.280499 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:02:44.280591 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:02:44.280673 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":55,"spinlock_wait_cycles":9856,"thread_start_us":98,"threads_started":1}
I20260927 15:02:44.281286 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.276471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35525 (local address 127.0.0.1:33796)
0927 15:02:44.276913 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.276929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:44.277030 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:44.277687 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:44.277692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:44.277716 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:44.277887 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:44.277896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.278987 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.278991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:44.279784 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:44.279793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.280756 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.280760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:44.280858 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:44.280900 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:02:44.281023 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:02:44.281103 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260927 15:02:44.281373 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.276666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.0.98.61:35525)
0927 15:02:44.277364 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:44.277382 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:44.277396 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:44.277444 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:44.277450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:44.277533 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:44.278010 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:44.278133 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.278875 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.279920 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:44.280624 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:44.280758 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:44.280818 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:44.281157 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 15:02:44.281231 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":637,"spinlock_wait_cycles":3968,"thread_start_us":65,"threads_started":1}
I20260927 15:02:44.281325 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbb0b749831846ac8eeac62a393c7eac" is_pre_election: true
I20260927 15:02:44.281544 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2e6f6795b0b4b968e95ca789b89be9a in term 1.
I20260927 15:02:44.281791 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb0b749831846ac8eeac62a393c7eac" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" is_pre_election: true
I20260927 15:02:44.281934 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb0b749831846ac8eeac62a393c7eac in term 1.
I20260927 15:02:44.281941 7086 leader_election.cc:302] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f2e6f6795b0b4b968e95ca789b89be9a, fbb0b749831846ac8eeac62a393c7eac; no voters: daf76d75dba64707a8ccd1ef2fa9a0c8
I20260927 15:02:44.282212 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:02:44.282264 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader daf76d75dba64707a8ccd1ef2fa9a0c8)
I20260927 15:02:44.282317 7809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:02:44.282435 7273 leader_election.cc:302] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f2e6f6795b0b4b968e95ca789b89be9a, fbb0b749831846ac8eeac62a393c7eac; no voters: daf76d75dba64707a8ccd1ef2fa9a0c8
I20260927 15:02:44.282651 7810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:02:44.282691 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Starting leader election (detected failure of leader daf76d75dba64707a8ccd1ef2fa9a0c8)
I20260927 15:02:44.282717 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:02:44.283573 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:44.283689 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:44.283993 7810 leader_election.cc:288] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 2 election: Requested vote from peers daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127), f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61:35525)
I20260927 15:02:44.283993 7809 leader_election.cc:288] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [CANDIDATE]: Term 2 election: Requested vote from peers daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127), fbb0b749831846ac8eeac62a393c7eac (127.0.98.60:46561)
I20260927 15:02:44.284384 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.284079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33840)
0927 15:02:44.284235 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.284290 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 15:02:44.284439 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb0b749831846ac8eeac62a393c7eac" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
I20260927 15:02:44.284550 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbb0b749831846ac8eeac62a393c7eac"
I20260927 15:02:44.284606 7105 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fbb0b749831846ac8eeac62a393c7eac in current term 2: Already voted for candidate f2e6f6795b0b4b968e95ca789b89be9a in this term.
I20260927 15:02:44.284730 7296 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f2e6f6795b0b4b968e95ca789b89be9a in current term 2: Already voted for candidate fbb0b749831846ac8eeac62a393c7eac in this term.
I20260927 15:02:44.285089 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.284890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33842)
0927 15:02:44.284973 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.285024 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 15:02:44.285161 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.284080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33838)
0927 15:02:44.284668 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.285091 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519}
W20260927 15:02:44.285211 7273 leader_election.cc:334] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:44.285298 7273 leader_election.cc:302] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb0b749831846ac8eeac62a393c7eac; no voters: daf76d75dba64707a8ccd1ef2fa9a0c8, f2e6f6795b0b4b968e95ca789b89be9a
I20260927 15:02:44.285419 7810 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 15:02:44.286008 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.285741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:33856)
0927 15:02:44.285875 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.285928 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 15:02:44.286188 7089 leader_election.cc:334] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:44.286247 7089 leader_election.cc:302] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2e6f6795b0b4b968e95ca789b89be9a; no voters: daf76d75dba64707a8ccd1ef2fa9a0c8, fbb0b749831846ac8eeac62a393c7eac
I20260927 15:02:44.286401 7809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 15:02:44.764357 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.763760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:34575)
0927 15:02:44.764164 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.764245 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":72,"threads_started":1}
I20260927 15:02:44.764827 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.763870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:52803)
0927 15:02:44.764654 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.764732 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":64,"threads_started":1}
I20260927 15:02:44.765436 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:44.765116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:60181)
0927 15:02:44.765320 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:44.765366 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":52,"threads_started":1}
I20260927 15:02:45.765636 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:45.764979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:36757)
0927 15:02:45.765433 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:45.765506 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":74,"threads_started":1}
I20260927 15:02:45.765971 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:45.765490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:45893)
0927 15:02:45.765814 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:45.765872 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":61,"threads_started":1}
I20260927 15:02:45.766546 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:45.766168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:52433)
0927 15:02:45.766389 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:45.766448 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":54,"threads_started":1}
I20260927 15:02:45.949922 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:45.950109 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:45.950637 7825 leader_election.cc:288] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127), f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61:35525)
I20260927 15:02:45.951201 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:45.950821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:34068)
0927 15:02:45.951031 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:45.951101 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260927 15:02:45.951426 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb0b749831846ac8eeac62a393c7eac" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" is_pre_election: true
I20260927 15:02:45.951613 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb0b749831846ac8eeac62a393c7eac in term 2.
I20260927 15:02:45.952010 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:45.951760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:34070)
0927 15:02:45.951899 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:45.951951 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:02:45.951987 7273 leader_election.cc:302] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2e6f6795b0b4b968e95ca789b89be9a, fbb0b749831846ac8eeac62a393c7eac; no voters:
I20260927 15:02:45.952178 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 15:02:45.952188 7273 leader_election.cc:334] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:45.952251 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:45.952292 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:02:45.953293 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:45.953603 7825 leader_election.cc:288] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 3 election: Requested vote from peers daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127), f2e6f6795b0b4b968e95ca789b89be9a (127.0.98.61:35525)
I20260927 15:02:45.953924 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:45.953697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:34086)
0927 15:02:45.953799 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:45.953852 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 15:02:45.954111 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb0b749831846ac8eeac62a393c7eac" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f2e6f6795b0b4b968e95ca789b89be9a"
I20260927 15:02:45.954238 7105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:02:45.954480 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:45.954314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:34100)
0927 15:02:45.954391 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:45.954437 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 15:02:45.954581 7273 leader_election.cc:334] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
I20260927 15:02:45.955220 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb0b749831846ac8eeac62a393c7eac in term 3.
I20260927 15:02:45.955507 7273 leader_election.cc:302] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f2e6f6795b0b4b968e95ca789b89be9a, fbb0b749831846ac8eeac62a393c7eac; no voters: daf76d75dba64707a8ccd1ef2fa9a0c8
I20260927 15:02:45.955719 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:02:45.955896 7825 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [term 3 LEADER]: Becoming Leader. State: Replica: fbb0b749831846ac8eeac62a393c7eac, State: Running, Role: LEADER
I20260927 15:02:45.956163 7825 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } }
I20260927 15:02:45.957496 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb0b749831846ac8eeac62a393c7eac. Latest consensus state: current_term: 3 leader_uuid: "fbb0b749831846ac8eeac62a393c7eac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:45.957654 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:45.957985 7830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:45.960058 7830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:45.960392 7830 catalog_manager.cc:1259] Loaded cluster ID: 4911026bf1eb4171bc627eb4ce0f4c46
I20260927 15:02:45.960434 7830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:45.960933 7830 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:45.961263 7830 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Loaded TSK: 0
I20260927 15:02:45.961962 7830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:02:46.022385 7286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:46.025483 7286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:46.028589 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.028302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:34112)
0927 15:02:46.028437 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.028508 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 15:02:46.028887 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [term 3 FOLLOWER]: Refusing update from remote peer fbb0b749831846ac8eeac62a393c7eac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 15:02:46.029287 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.029051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.0.1:34126)
0927 15:02:46.029155 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.029228 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:02:46.029383 7273 consensus_peers.cc:487] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac -> Peer daf76d75dba64707a8ccd1ef2fa9a0c8 (127.0.98.62:41127): Couldn't send request to peer daf76d75dba64707a8ccd1ef2fa9a0c8. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:02:46.029404 7825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 15:02:46.030853 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb0b749831846ac8eeac62a393c7eac. Latest consensus state: current_term: 3 leader_uuid: "fbb0b749831846ac8eeac62a393c7eac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:46.031015 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:02:46.032130 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fbb0b749831846ac8eeac62a393c7eac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:46.032254 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:46.032529 7833 mvcc.cc:205] Tried to move back new op lower bound from 7333975515248230400 to 7333975514958811136. Current Snapshot: MvccSnapshot[applied={T|T < 7333975515248230400}]
I20260927 15:02:46.032657 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fbb0b749831846ac8eeac62a393c7eac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "daf76d75dba64707a8ccd1ef2fa9a0c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41127 } } peers { permanent_uuid: "f2e6f6795b0b4b968e95ca789b89be9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "fbb0b749831846ac8eeac62a393c7eac" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46561 } attrs { promote: false } } }
I20260927 15:02:46.032766 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6f6795b0b4b968e95ca789b89be9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:02:46.043187 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.038805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46395 (local address 127.0.0.1:50612)
0927 15:02:46.039354 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.039368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:46.039460 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:46.039841 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:46.039844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:46.039856 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:46.039940 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.039949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.040985 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.040989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.041846 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.041856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.042741 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.042744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.042834 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.042858 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:02:46.042961 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:02:46.043038 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":256,"thread_start_us":97,"threads_started":1}
I20260927 15:02:46.043187 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.038926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.0.98.3:46395)
0927 15:02:46.039177 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:46.039190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:46.039452 (+ 262us) server_negotiation.cc:407] Connection header received
0927 15:02:46.039547 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:46.039591 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:46.039651 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:46.039734 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:46.040055 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.040866 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.041988 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.042613 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.042731 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.042799 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:46.042979 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 15:02:46.043046 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260927 15:02:46.043743 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.038234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34969 (local address 127.0.0.1:49044)
0927 15:02:46.038346 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.038361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:46.038443 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:46.039989 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:46.039992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:46.040007 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:46.040095 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.040100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.040882 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.040889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.042422 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.042432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.043336 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.043340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.043450 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.043475 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:02:46.043538 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:02:46.043607 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":178}
I20260927 15:02:46.043828 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.038379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.98.2:34969)
0927 15:02:46.039748 (+ 1369us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:46.039753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:46.039770 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:02:46.039823 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:46.039826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:46.039871 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:46.039980 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:46.040226 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.040764 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.042564 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.043199 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.043385 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.043580 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:46.043617 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 15:02:46.043690 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":71,"threads_started":1}
I20260927 15:02:46.043879 7674 tablet_service.cc:1467] Processing CreateTablet for tablet daa5e77012d847b5b5eb469964c14b8a (DEFAULT_TABLE table=test-table [id=12e893ff58c14dee8bdf40482dfb4954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:46.044209 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa5e77012d847b5b5eb469964c14b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:46.044317 7536 tablet_service.cc:1467] Processing CreateTablet for tablet daa5e77012d847b5b5eb469964c14b8a (DEFAULT_TABLE table=test-table [id=12e893ff58c14dee8bdf40482dfb4954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:46.044495 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.038805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38235 (local address 127.0.0.1:40474)
0927 15:02:46.039100 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.039116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:46.039209 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:46.039729 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:46.039732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:46.039747 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:46.039854 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.039861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.040942 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.040945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.042424 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.042432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.043336 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.043340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.044173 (+ 833us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.044197 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:02:46.044275 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:02:46.044352 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":187,"thread_start_us":106,"threads_started":1}
I20260927 15:02:46.044574 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa5e77012d847b5b5eb469964c14b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:46.045055 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.039260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.0.98.1:38235)
0927 15:02:46.039491 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:46.039495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:46.039509 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:46.039553 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:46.039556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:46.039601 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:46.039966 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:46.040086 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.040851 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.042564 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.043195 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.044791 (+ 1596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.044840 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:46.044870 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 15:02:46.044929 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260927 15:02:46.045600 7399 tablet_service.cc:1467] Processing CreateTablet for tablet daa5e77012d847b5b5eb469964c14b8a (DEFAULT_TABLE table=test-table [id=12e893ff58c14dee8bdf40482dfb4954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:46.045897 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa5e77012d847b5b5eb469964c14b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:46.047012 7844 tablet_bootstrap.cc:492] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: Bootstrap starting.
I20260927 15:02:46.048026 7844 tablet_bootstrap.cc:654] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:46.048321 7845 tablet_bootstrap.cc:492] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2: Bootstrap starting.
I20260927 15:02:46.048651 7843 tablet_bootstrap.cc:492] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Bootstrap starting.
I20260927 15:02:46.049196 7845 tablet_bootstrap.cc:654] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:46.049664 7843 tablet_bootstrap.cc:654] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:46.050254 7844 tablet_bootstrap.cc:492] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: No bootstrap required, opened a new log
I20260927 15:02:46.050364 7844 ts_tablet_manager.cc:1397] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:46.050608 7845 tablet_bootstrap.cc:492] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2: No bootstrap required, opened a new log
I20260927 15:02:46.050684 7845 ts_tablet_manager.cc:1397] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 15:02:46.050788 7844 raft_consensus.cc:348] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.050912 7844 raft_consensus.cc:374] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:46.050997 7844 raft_consensus.cc:729] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee668e13116e4035b8861fd5af19cfa3, State: Initialized, Role: FOLLOWER
I20260927 15:02:46.051152 7844 consensus_queue.cc:260] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.051172 7845 raft_consensus.cc:348] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.051312 7845 raft_consensus.cc:374] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:46.051357 7845 raft_consensus.cc:729] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58a3ca2f5f74e0faed62bbb2bd313e2, State: Initialized, Role: FOLLOWER
I20260927 15:02:46.051519 7844 ts_tablet_manager.cc:1428] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 15:02:46.051538 7845 consensus_queue.cc:260] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.051997 7845 ts_tablet_manager.cc:1428] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 15:02:46.052184 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.051850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:36717)
0927 15:02:46.052048 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.052108 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 15:02:46.052796 7843 tablet_bootstrap.cc:492] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: No bootstrap required, opened a new log
I20260927 15:02:46.052870 7843 ts_tablet_manager.cc:1397] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:02:46.052958 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.052700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:40445)
0927 15:02:46.052843 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.052895 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 15:02:46.053304 7843 raft_consensus.cc:348] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.053407 7843 raft_consensus.cc:374] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:46.053443 7843 raft_consensus.cc:729] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef32965fdc614ddb9312d28c310ba524, State: Initialized, Role: FOLLOWER
I20260927 15:02:46.053596 7843 consensus_queue.cc:260] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.053839 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.053450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:49709)
0927 15:02:46.053705 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.053751 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260927 15:02:46.054507 7843 ts_tablet_manager.cc:1428] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:46.054564 7465 heartbeater.cc:503] Master 127.0.98.60:46561 was elected leader, sending a full tablet report...
I20260927 15:02:46.055230 7602 heartbeater.cc:503] Master 127.0.98.60:46561 was elected leader, sending a full tablet report...
I20260927 15:02:46.055928 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.055630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:35237)
0927 15:02:46.055816 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.055865 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 15:02:46.056275 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.055658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:56861)
0927 15:02:46.056134 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.056188 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
I20260927 15:02:46.056978 7740 heartbeater.cc:503] Master 127.0.98.60:46561 was elected leader, sending a full tablet report...
I20260927 15:02:46.057403 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.056979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:38819)
0927 15:02:46.057266 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.057326 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260927 15:02:46.101163 7849 raft_consensus.cc:482] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:46.101308 7849 raft_consensus.cc:504] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.102885 7849 leader_election.cc:288] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969), ef32965fdc614ddb9312d28c310ba524 (127.0.98.3:46395)
I20260927 15:02:46.106621 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.103216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58309 (local address 127.0.98.2:34969)
0927 15:02:46.103340 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:46.103344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:46.103460 (+ 116us) server_negotiation.cc:407] Connection header received
0927 15:02:46.103564 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:46.103567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:46.103616 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:46.103698 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:46.104025 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.104549 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.105614 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.106024 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.106109 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.106207 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:46.106402 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 15:02:46.106485 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:02:46.106628 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.102772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46395 (local address 127.0.98.1:52487)
0927 15:02:46.103107 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.103124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:46.103230 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:46.103565 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:46.103568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:46.103589 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:46.103723 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.103730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.104529 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.104532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.105429 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.105438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.106208 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.106212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.106305 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.106322 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:02:46.106418 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:02:46.106484 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":3}
I20260927 15:02:46.106639 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.103006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52487 (local address 127.0.98.3:46395)
0927 15:02:46.103134 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:46.103138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:46.103320 (+ 182us) server_negotiation.cc:407] Connection header received
0927 15:02:46.103377 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:46.103382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:46.103444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:46.103519 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:46.103842 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.104409 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.105559 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.106057 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.106184 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.106247 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:46.106446 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 15:02:46.106517 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:02:46.106647 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.102769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34969 (local address 127.0.98.1:58309)
0927 15:02:46.103363 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.103375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:46.103462 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:46.103720 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:46.103722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:46.103755 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:46.103884 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.103890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.104669 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.104672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.105463 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:46.105472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.106174 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.106178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:46.106278 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.106303 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:02:46.106363 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:02:46.106452 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":113,"thread_start_us":111,"threads_started":1}
I20260927 15:02:46.107216 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa5e77012d847b5b5eb469964c14b8a" candidate_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef32965fdc614ddb9312d28c310ba524" is_pre_election: true
I20260927 15:02:46.107256 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa5e77012d847b5b5eb469964c14b8a" candidate_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee668e13116e4035b8861fd5af19cfa3" is_pre_election: true
I20260927 15:02:46.107400 7556 raft_consensus.cc:2454] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58a3ca2f5f74e0faed62bbb2bd313e2 in term 0.
I20260927 15:02:46.107448 7694 raft_consensus.cc:2454] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58a3ca2f5f74e0faed62bbb2bd313e2 in term 0.
I20260927 15:02:46.107761 7352 leader_election.cc:302] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e58a3ca2f5f74e0faed62bbb2bd313e2, ee668e13116e4035b8861fd5af19cfa3; no voters:
I20260927 15:02:46.108017 7849 raft_consensus.cc:2781] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:46.108088 7849 raft_consensus.cc:482] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:46.108147 7849 raft_consensus.cc:3037] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:46.109129 7849 raft_consensus.cc:504] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.109508 7849 leader_election.cc:288] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [CANDIDATE]: Term 1 election: Requested vote from peers ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969), ef32965fdc614ddb9312d28c310ba524 (127.0.98.3:46395)
I20260927 15:02:46.109861 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa5e77012d847b5b5eb469964c14b8a" candidate_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef32965fdc614ddb9312d28c310ba524"
I20260927 15:02:46.109861 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa5e77012d847b5b5eb469964c14b8a" candidate_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee668e13116e4035b8861fd5af19cfa3"
I20260927 15:02:46.109973 7694 raft_consensus.cc:3037] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:46.109997 7556 raft_consensus.cc:3037] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:46.111030 7556 raft_consensus.cc:2454] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58a3ca2f5f74e0faed62bbb2bd313e2 in term 1.
I20260927 15:02:46.111030 7694 raft_consensus.cc:2454] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58a3ca2f5f74e0faed62bbb2bd313e2 in term 1.
I20260927 15:02:46.111311 7355 leader_election.cc:302] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e58a3ca2f5f74e0faed62bbb2bd313e2, ef32965fdc614ddb9312d28c310ba524; no voters:
I20260927 15:02:46.111496 7849 raft_consensus.cc:2781] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:46.111733 7849 raft_consensus.cc:686] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 LEADER]: Becoming Leader. State: Replica: e58a3ca2f5f74e0faed62bbb2bd313e2, State: Running, Role: LEADER
I20260927 15:02:46.112002 7849 consensus_queue.cc:237] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } }
I20260927 15:02:46.112418 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.112166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:55821)
0927 15:02:46.112305 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.112353 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 15:02:46.114460 7284 catalog_manager.cc:5495] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58a3ca2f5f74e0faed62bbb2bd313e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 } health_report { overall_health: UNKNOWN } } }
I20260927 15:02:46.127300 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.123591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.0.98.1:38235)
0927 15:02:46.123725 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:46.123728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:46.123741 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:02:46.123812 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:46.123814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:46.123869 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:46.123965 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:46.124187 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.124726 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.125894 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:46.126109 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:46.126226 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:46.126485 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:46.126919 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:46.127051 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:46.127111 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 15:02:46.127167 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:02:46.130185 7556 raft_consensus.cc:1261] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Refusing update from remote peer e58a3ca2f5f74e0faed62bbb2bd313e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:46.130402 7694 raft_consensus.cc:1261] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 1 FOLLOWER]: Refusing update from remote peer e58a3ca2f5f74e0faed62bbb2bd313e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:46.130705 7849 consensus_queue.cc:1035] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee668e13116e4035b8861fd5af19cfa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:46.130959 7853 consensus_queue.cc:1035] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef32965fdc614ddb9312d28c310ba524" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:46.132742 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.132426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:55259)
0927 15:02:46.132598 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.132669 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 15:02:46.133399 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.133133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:46075)
0927 15:02:46.133271 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.133331 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:02:46.135337 7743 tablet.cc:2369] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:46.136804 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.136516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:57215)
0927 15:02:46.136674 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.136735 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 15:02:46.137308 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.136997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.2:35543)
0927 15:02:46.137182 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.137236 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 15:02:46.137547 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.137306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.1:59537)
0927 15:02:46.137437 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.137483 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:02:46.139747 7859 mvcc.cc:205] Tried to move back new op lower bound from 7333975515663290368 to 7333975515597484032. Current Snapshot: MvccSnapshot[applied={T|T < 7333975515663290368}]
I20260927 15:02:46.141000 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.140759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41127 (local address 127.0.98.3:43857)
0927 15:02:46.140874 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.140931 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:02:46.146929 7286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:46.147116 7286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:46.151278 7286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbb0b749831846ac8eeac62a393c7eac: Sending DeleteTablet for 3 replicas of tablet daa5e77012d847b5b5eb469964c14b8a
I20260927 15:02:46.151943 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet daa5e77012d847b5b5eb469964c14b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:46 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40474
I20260927 15:02:46.151929 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet daa5e77012d847b5b5eb469964c14b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:46 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49044
I20260927 15:02:46.152268 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet daa5e77012d847b5b5eb469964c14b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:46 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50612
I20260927 15:02:46.152441 7867 tablet_replica.cc:330] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2: stopping tablet replica
I20260927 15:02:46.152597 7867 raft_consensus.cc:2229] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:46.152643 7869 tablet_replica.cc:330] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: stopping tablet replica
I20260927 15:02:46.152854 7869 raft_consensus.cc:2229] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:46.152925 7867 raft_consensus.cc:2258] T daa5e77012d847b5b5eb469964c14b8a P e58a3ca2f5f74e0faed62bbb2bd313e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:46.153074 7869 raft_consensus.cc:2258] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:46.153183 7868 tablet_replica.cc:330] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: stopping tablet replica
I20260927 15:02:46.153304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 7247
I20260927 15:02:46.153910 7868 raft_consensus.cc:2229] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:46.154053 7868 raft_consensus.cc:2258] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:46.155017 7868 ts_tablet_manager.cc:1905] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:46.156401 7869 ts_tablet_manager.cc:1905] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:46.157485 7868 ts_tablet_manager.cc:1918] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:46.157568 7868 log.cc:1196] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/daa5e77012d847b5b5eb469964c14b8a
I20260927 15:02:46.157866 7868 ts_tablet_manager.cc:1939] T daa5e77012d847b5b5eb469964c14b8a P ee668e13116e4035b8861fd5af19cfa3: Deleting consensus metadata
I20260927 15:02:46.158602 7270 catalog_manager.cc:4841] TS ee668e13116e4035b8861fd5af19cfa3 (127.0.98.2:34969): tablet daa5e77012d847b5b5eb469964c14b8a (table test-table [id=12e893ff58c14dee8bdf40482dfb4954]) successfully deleted
I20260927 15:02:46.158969 7869 ts_tablet_manager.cc:1918] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:46.159031 7869 log.cc:1196] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/daa5e77012d847b5b5eb469964c14b8a
I20260927 15:02:46.159291 7869 ts_tablet_manager.cc:1939] T daa5e77012d847b5b5eb469964c14b8a P ef32965fdc614ddb9312d28c310ba524: Deleting consensus metadata
I20260927 15:02:46.159859 7273 catalog_manager.cc:4841] TS ef32965fdc614ddb9312d28c310ba524 (127.0.98.3:46395): tablet daa5e77012d847b5b5eb469964c14b8a (table test-table [id=12e893ff58c14dee8bdf40482dfb4954]) successfully deleted
I20260927 15:02:46.161082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 7473
I20260927 15:02:46.161574 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:46.161258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38235 (local address 127.0.0.1:40488)
0927 15:02:46.161429 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:46.161482 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 15:02:46.161856 7271 catalog_manager.cc:4584] TS e58a3ca2f5f74e0faed62bbb2bd313e2 (127.0.98.1:38235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet daa5e77012d847b5b5eb469964c14b8a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38235: connect: Connection refused (error 111)
I20260927 15:02:46.166664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 7609
I20260927 15:02:46.173403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 6966
I20260927 15:02:46.179008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 7074
2026-09-27T15:02:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6931 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:46 dist-test-slave-ml7t krb5kdc[7872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:46 dist-test-slave-ml7t krb5kdc[7872](info): set up 2 sockets
Sep 27 15:02:46 dist-test-slave-ml7t krb5kdc[7872](info): commencing operation
krb5kdc: starting...
W20260927 15:02:47.246495 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:47 dist-test-slave-ml7t krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 15:02:47.373106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:47.480427 7885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:47.480693 7885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:47.480732 7885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:47.484650 7885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:47.484704 7885 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:02:47.484720 7885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:47.484738 7885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:47.484754 7885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:47.484768 7885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:47.489838 7885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43961
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:47.490759 7885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:47.491708 7885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:47.499816 7891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:47.499859 7893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:47.499820 7890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:47.500176 7885 server_base.cc:1034] running on GCE node
I20260927 15:02:47.500722 7885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:47.501883 7885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:47.503070 7885 hybrid_clock.cc:648] HybridClock initialized: now 1790521367503044 us; error 41 us; skew 500 ppm
Sep 27 15:02:47 dist-test-slave-ml7t krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521367, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:47.505986 7885 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 15:02:47.506808 7885 webserver.cc:460] Webserver started at http://127.0.98.62:33473/ using document root <none> and password file <none>
I20260927 15:02:47.507339 7885 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:47.507426 7885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:47.507666 7885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:47.509611 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "41f6ff9664134b4f9ff96687b0ed4485"
format_stamp: "Formatted at 2026-09-27 15:02:47 on dist-test-slave-ml7t"
server_key: "f82ca8a333e705d7ed443336d1196be0"
server_key_iv: "2b7e3a8574542852bef67f35294c1ea1"
server_key_version: "encryptionkey@0"
I20260927 15:02:47.510214 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "41f6ff9664134b4f9ff96687b0ed4485"
format_stamp: "Formatted at 2026-09-27 15:02:47 on dist-test-slave-ml7t"
server_key: "f82ca8a333e705d7ed443336d1196be0"
server_key_iv: "2b7e3a8574542852bef67f35294c1ea1"
server_key_version: "encryptionkey@0"
I20260927 15:02:47.513708 7885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:02:47.516009 7900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:47.517241 7885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:02:47.517396 7885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "41f6ff9664134b4f9ff96687b0ed4485"
format_stamp: "Formatted at 2026-09-27 15:02:47 on dist-test-slave-ml7t"
server_key: "f82ca8a333e705d7ed443336d1196be0"
server_key_iv: "2b7e3a8574542852bef67f35294c1ea1"
server_key_version: "encryptionkey@0"
I20260927 15:02:47.517508 7885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:47.530601 7885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:47.533813 7885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:47.533998 7885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:47.541846 7885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43961
I20260927 15:02:47.541864 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43961 every 8 connection(s)
I20260927 15:02:47.543072 7885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:47.545995 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:47.549134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 7885
I20260927 15:02:47.549278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:47.549590 392 external_mini_cluster.cc:1428] Setting key d206828919cd2ffdc76e191cfb3341ca
I20260927 15:02:47.552009 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Bootstrap starting.
I20260927 15:02:47.554494 7963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:47.555250 7963 log.cc:824] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:47.557446 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: No bootstrap required, opened a new log
Sep 27 15:02:47 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:47.562011 7963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f6ff9664134b4f9ff96687b0ed4485" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43961 } }
I20260927 15:02:47.562281 7963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:47.562340 7963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f6ff9664134b4f9ff96687b0ed4485, State: Initialized, Role: FOLLOWER
I20260927 15:02:47.562846 7963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f6ff9664134b4f9ff96687b0ed4485" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43961 } }
I20260927 15:02:47.562976 7963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:47.563064 7963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:47.563170 7963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:47.564129 7963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f6ff9664134b4f9ff96687b0ed4485" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43961 } }
I20260927 15:02:47.564478 7963 leader_election.cc:302] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f6ff9664134b4f9ff96687b0ed4485; no voters:
I20260927 15:02:47.564817 7963 leader_election.cc:288] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:47.564886 7968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:47.565019 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:47.551082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.0.98.62:43961)
0927 15:02:47.551509 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:47.551524 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:47.551559 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:02:47.552239 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:47.552257 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:47.552581 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:47.552913 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:47.553829 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.554956 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.556056 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.556742 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.556908 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:47.559719 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:47.559727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:47.559745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:47.559774 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:47.561738 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:47.562206 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:47.562212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:47.562218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:47.562361 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:47.562686 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:47.562692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:47.562694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:47.563240 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:47.563443 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:47.563793 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 15:02:47.564093 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":132,"threads_started":1}
I20260927 15:02:47.566219 7963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:47.565102 7968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [term 1 LEADER]: Becoming Leader. State: Replica: 41f6ff9664134b4f9ff96687b0ed4485, State: Running, Role: LEADER
I20260927 15:02:47.566782 7968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f6ff9664134b4f9ff96687b0ed4485" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43961 } }
I20260927 15:02:47.568624 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f6ff9664134b4f9ff96687b0ed4485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f6ff9664134b4f9ff96687b0ed4485" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43961 } } }
I20260927 15:02:47.568781 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f6ff9664134b4f9ff96687b0ed4485. Latest consensus state: current_term: 1 leader_uuid: "41f6ff9664134b4f9ff96687b0ed4485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f6ff9664134b4f9ff96687b0ed4485" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43961 } } }
I20260927 15:02:47.568920 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:47.568904 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485 [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:47.569345 7983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:47.572293 7983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:47.579035 7983 catalog_manager.cc:1347] Generated new cluster ID: c228bfb754df4aa3aabfdc56d59522e6
I20260927 15:02:47.579130 7983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:47.604625 7983 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:47.605631 7983 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:47.614923 7983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Generated new TSK 0
I20260927 15:02:47.615623 7983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 15:02:47.645182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43961
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:47.755609 7989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:47.755875 7989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:47.755923 7989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:47.759989 7989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:47.760052 7989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:47.760133 7989 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:47.760196 7989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:47.765441 7989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:47.766577 7989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:47.767696 7989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:47.777345 7989 server_base.cc:1034] running on GCE node
W20260927 15:02:47.777354 7997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:47.777366 7995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:47.777366 7994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:47.778074 7989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:47.778753 7989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:47.779930 7989 hybrid_clock.cc:648] HybridClock initialized: now 1790521367779893 us; error 49 us; skew 500 ppm
Sep 27 15:02:47 dist-test-slave-ml7t krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521367, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:47.782864 7989 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 15:02:47.783774 7989 webserver.cc:460] Webserver started at http://127.0.98.1:35895/ using document root <none> and password file <none>
I20260927 15:02:47.784356 7989 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:47.784452 7989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:47.784696 7989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:47.787011 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "d35c29d132e0439e87df1556a51e4e90"
format_stamp: "Formatted at 2026-09-27 15:02:47 on dist-test-slave-ml7t"
server_key: "d9a285b2eedcb136ff1909d1668e01d4"
server_key_iv: "3ec9ce14483cf99d8a6f6525e307c0f3"
server_key_version: "encryptionkey@0"
I20260927 15:02:47.787618 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d35c29d132e0439e87df1556a51e4e90"
format_stamp: "Formatted at 2026-09-27 15:02:47 on dist-test-slave-ml7t"
server_key: "d9a285b2eedcb136ff1909d1668e01d4"
server_key_iv: "3ec9ce14483cf99d8a6f6525e307c0f3"
server_key_version: "encryptionkey@0"
I20260927 15:02:47.791651 7989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:02:47.794272 8004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:47.795594 7989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:47.795773 7989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "d35c29d132e0439e87df1556a51e4e90"
format_stamp: "Formatted at 2026-09-27 15:02:47 on dist-test-slave-ml7t"
server_key: "d9a285b2eedcb136ff1909d1668e01d4"
server_key_iv: "3ec9ce14483cf99d8a6f6525e307c0f3"
server_key_version: "encryptionkey@0"
I20260927 15:02:47.795897 7989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:47.820353 7989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:47.823644 7989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:47.823859 7989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:47.825309 7989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:47.825388 7989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:47.825457 7989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:47.825518 7989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:02:47 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521367, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:47.836750 7989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39649
I20260927 15:02:47.837456 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39649 every 8 connection(s)
I20260927 15:02:47.837661 7989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:47.839388 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:47.826296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33103 (local address 127.0.98.62:43961)
0927 15:02:47.826421 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:47.826425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:47.827533 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 15:02:47.828559 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:47.828564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:47.828664 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:47.828733 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:47.830463 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.831142 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.832493 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.832779 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.832905 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:47.835701 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:47.835704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:47.835707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:47.835739 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:47.837242 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:47.837874 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:47.837878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:47.837880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:47.837972 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:47.838378 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:47.838382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:47.838383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:47.838629 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:47.838711 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:47.839172 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 15:02:47.839271 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:02:47.840191 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:47.826850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43961 (local address 127.0.98.1:33103)
0927 15:02:47.827365 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:47.827409 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:47.828267 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:47.829246 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:47.829259 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:47.829743 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:47.830305 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:47.830319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.831265 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.831268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:47.832369 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:47.832378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.832961 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.832965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:47.833166 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:47.833848 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:47.833865 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:47.835519 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:47.837361 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:47.837366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:47.837383 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:47.837722 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:47.838095 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:47.838098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:47.838100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:47.838258 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:47.838742 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:47.838747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:47.838959 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 15:02:47.839111 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 15:02:47.839348 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":146,"threads_started":1}
I20260927 15:02:47.841466 8120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43961
I20260927 15:02:47.841642 8120 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:47.841862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 7989
I20260927 15:02:47.841987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:47.842275 8120 heartbeater.cc:511] Master 127.0.98.62:43961 requested a full tablet report, sending...
I20260927 15:02:47.842298 392 external_mini_cluster.cc:1428] Setting key f388af98c4f69b1cd53323fb4ca42bfe
I20260927 15:02:47.843842 7917 ts_manager.cc:194] Registered new tserver with Master: d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649)
I20260927 15:02:47.845679 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33103
I20260927 15:02:47.852095 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:47.843972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43961 (local address 127.0.98.1:53153)
0927 15:02:47.844317 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:47.844329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:47.844444 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:47.844669 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:47.844673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:47.844891 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:47.844987 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:47.844992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.845977 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.845981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:47.846928 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:47.846937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.847398 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.847401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:47.847493 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:47.848051 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:47.848071 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:47.848535 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:47.850647 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:47.850650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:47.850652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:47.850857 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:47.851163 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:47.851165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:47.851166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:47.851238 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:47.851652 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:47.851654 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:47.851726 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:02:47.851810 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:02:47.851919 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260927 15:02:47.852222 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:47.844022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53153 (local address 127.0.98.62:43961)
0927 15:02:47.844176 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:47.844183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:47.844405 (+ 222us) server_negotiation.cc:407] Connection header received
0927 15:02:47.844521 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:47.844524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:47.844569 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:47.844655 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:47.845102 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.845819 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.847042 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:47.847284 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:47.847394 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:47.848663 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:47.848667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:47.848669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:47.848688 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:47.850507 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:47.850955 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:47.850958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:47.850962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:47.851054 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:47.851326 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:47.851329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:47.851331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:47.851553 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:47.851665 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:47.851904 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 15:02:47.852019 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 15:02:47.873878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43961
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:47.989152 8129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:47.989478 8129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:47.989557 8129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:47.993686 8129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:47.993821 8129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:47.993897 8129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:47.993983 8129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:48.000078 8129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:48.001477 8129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:48.003036 8129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:48.014118 8137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:48.014119 8134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:48.014164 8135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:48.014443 8129 server_base.cc:1034] running on GCE node
I20260927 15:02:48.014883 8129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:48.015630 8129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:48.016871 8129 hybrid_clock.cc:648] HybridClock initialized: now 1790521368016842 us; error 52 us; skew 500 ppm
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:48.020102 8129 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 15:02:48.021337 8129 webserver.cc:460] Webserver started at http://127.0.98.2:39693/ using document root <none> and password file <none>
I20260927 15:02:48.022130 8129 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:48.022233 8129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:48.022488 8129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:48.024765 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "c17e138e781348d885de9934dde1a0fd"
format_stamp: "Formatted at 2026-09-27 15:02:48 on dist-test-slave-ml7t"
server_key: "a11c1df9738d84fecfa0150749dd58e9"
server_key_iv: "5a2a9196af997cb0c736833cff5c8f68"
server_key_version: "encryptionkey@0"
I20260927 15:02:48.025434 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c17e138e781348d885de9934dde1a0fd"
format_stamp: "Formatted at 2026-09-27 15:02:48 on dist-test-slave-ml7t"
server_key: "a11c1df9738d84fecfa0150749dd58e9"
server_key_iv: "5a2a9196af997cb0c736833cff5c8f68"
server_key_version: "encryptionkey@0"
I20260927 15:02:48.029596 8129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 15:02:48.032335 8144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:48.033692 8129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:48.033855 8129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "c17e138e781348d885de9934dde1a0fd"
format_stamp: "Formatted at 2026-09-27 15:02:48 on dist-test-slave-ml7t"
server_key: "a11c1df9738d84fecfa0150749dd58e9"
server_key_iv: "5a2a9196af997cb0c736833cff5c8f68"
server_key_version: "encryptionkey@0"
I20260927 15:02:48.033983 8129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:48.081389 8129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:48.085147 8129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:48.085541 8129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:48.087275 8129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:48.087373 8129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:48.087450 8129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:48.087513 8129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:48.100183 8129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33505
I20260927 15:02:48.100314 8259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33505 every 8 connection(s)
I20260927 15:02:48.101616 8129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:48.104233 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.088223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46073 (local address 127.0.98.62:43961)
0927 15:02:48.088383 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.088388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.089359 (+ 971us) server_negotiation.cc:407] Connection header received
0927 15:02:48.090483 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.090487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.090543 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.090603 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.092178 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.093061 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.094683 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.094979 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.095112 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.098630 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:48.098635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:48.098638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:48.098677 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:48.100981 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.101908 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.101912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.101914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.102015 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.102523 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.102530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.102532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.102814 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:48.103124 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.103850 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.104018 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:02:48.105197 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.088680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43961 (local address 127.0.98.2:46073)
0927 15:02:48.089168 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.089220 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.090209 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.091097 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.091110 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.091516 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:48.092014 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.092027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.093187 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.093193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.094544 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.094555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.095170 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.095174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.095364 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.096165 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:48.096193 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:48.098344 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:48.101271 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.101278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.101296 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.101739 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.102136 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.102140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.102143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.102357 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.103161 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:48.103169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:48.103501 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.103762 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.104177 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":138,"threads_started":1}
I20260927 15:02:48.106678 8260 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43961
I20260927 15:02:48.106871 8260 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:48.107431 8260 heartbeater.cc:511] Master 127.0.98.62:43961 requested a full tablet report, sending...
I20260927 15:02:48.108657 7916 ts_manager.cc:194] Registered new tserver with Master: c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505)
I20260927 15:02:48.109546 7916 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46073
I20260927 15:02:48.111610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 8129
I20260927 15:02:48.111755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:48.112090 392 external_mini_cluster.cc:1428] Setting key 8b3637d359a7aed4e58a3f2d63f772c3
I20260927 15:02:48.118340 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.109264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49597 (local address 127.0.98.62:43961)
0927 15:02:48.109389 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.109392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.110196 (+ 804us) server_negotiation.cc:407] Connection header received
0927 15:02:48.110330 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.110333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.110381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.110473 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.110942 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.111675 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.112740 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.113069 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.113160 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.114543 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:48.114546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:48.114548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:48.114570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:48.116342 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.116883 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.116889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.116891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.116985 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.117313 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.117317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.117318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.117559 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:48.117683 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.118046 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.118167 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":49}
I20260927 15:02:48.118353 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.109165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43961 (local address 127.0.98.2:49597)
0927 15:02:48.110077 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.110096 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.110222 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.110498 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.110500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.110708 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:48.110817 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.110823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.111832 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.111838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.112591 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.112616 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.113192 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.113198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.113319 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.113990 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:48.114001 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:48.114357 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:48.116534 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.116537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.116540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.116769 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.117102 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.117105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.117108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.117206 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.117671 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:48.117675 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:48.117829 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.117982 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.118143 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 15:02:48.141741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43961
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:48.256599 8269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:48.256887 8269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:48.256932 8269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:48.261089 8269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:48.261160 8269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:48.261199 8269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:02:48.261279 8269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:48.266726 8269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:48.267961 8269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:48.269043 8269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:48.278403 8277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:48.278386 8274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:48.278385 8275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:48.278566 8269 server_base.cc:1034] running on GCE node
I20260927 15:02:48.279160 8269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:48.279781 8269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:48.280944 8269 hybrid_clock.cc:648] HybridClock initialized: now 1790521368280933 us; error 26 us; skew 500 ppm
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:48.283679 8269 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 15:02:48.284521 8269 webserver.cc:460] Webserver started at http://127.0.98.3:44541/ using document root <none> and password file <none>
I20260927 15:02:48.285036 8269 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:48.285091 8269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:48.285269 8269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:48.287334 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "36b2f430de574935bd3cd269226b726f"
format_stamp: "Formatted at 2026-09-27 15:02:48 on dist-test-slave-ml7t"
server_key: "4ba1b962c476a27bc7a60f77e5144402"
server_key_iv: "eb437c7b389959da09a276d24c4d6354"
server_key_version: "encryptionkey@0"
I20260927 15:02:48.287843 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "36b2f430de574935bd3cd269226b726f"
format_stamp: "Formatted at 2026-09-27 15:02:48 on dist-test-slave-ml7t"
server_key: "4ba1b962c476a27bc7a60f77e5144402"
server_key_iv: "eb437c7b389959da09a276d24c4d6354"
server_key_version: "encryptionkey@0"
I20260927 15:02:48.291455 8269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 15:02:48.293929 8284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:48.295136 8269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:48.295264 8269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "36b2f430de574935bd3cd269226b726f"
format_stamp: "Formatted at 2026-09-27 15:02:48 on dist-test-slave-ml7t"
server_key: "4ba1b962c476a27bc7a60f77e5144402"
server_key_iv: "eb437c7b389959da09a276d24c4d6354"
server_key_version: "encryptionkey@0"
I20260927 15:02:48.295377 8269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:48.332569 8269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:48.335958 8269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:48.336162 8269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:48.337625 8269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:48.337697 8269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:48.337800 8269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:48.337869 8269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:48.348889 8269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41659
I20260927 15:02:48.349593 8399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41659 every 8 connection(s)
I20260927 15:02:48.349913 8269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:48.351665 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.338627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58609 (local address 127.0.98.62:43961)
0927 15:02:48.338792 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.338797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.339716 (+ 919us) server_negotiation.cc:407] Connection header received
0927 15:02:48.340925 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.340932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.341002 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.341130 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.342624 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.343221 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.344563 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.344873 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.344995 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.347894 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:48.347896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:48.347898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:48.347922 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:48.349478 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.350097 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.350101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.350108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.350183 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.350595 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.350599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.350601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.350836 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:48.350939 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.351446 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.351543 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:02:48.352512 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.339069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43961 (local address 127.0.98.3:58609)
0927 15:02:48.339527 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.339594 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.340496 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.341338 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.341348 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.341835 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:48.342471 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.342486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.343343 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.343347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.344435 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.344445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.345037 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.345041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.345229 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.345852 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:48.345871 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:48.347731 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:48.349604 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.349610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.349624 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.349995 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.350282 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.350286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.350288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.350461 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.350945 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:48.350950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:48.351218 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.351365 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.351597 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":141,"threads_started":1}
I20260927 15:02:48.353720 8400 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43961
I20260927 15:02:48.353991 8400 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:48.354552 8400 heartbeater.cc:511] Master 127.0.98.62:43961 requested a full tablet report, sending...
I20260927 15:02:48.355645 7917 ts_manager.cc:194] Registered new tserver with Master: 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659)
I20260927 15:02:48.356395 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58609
I20260927 15:02:48.358737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 8269
I20260927 15:02:48.358876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:48.359174 392 external_mini_cluster.cc:1428] Setting key 618b9348ee5c8851ed8c255dcf3e6e28
I20260927 15:02:48.361133 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:02:48.364614 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.356376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38061 (local address 127.0.98.62:43961)
0927 15:02:48.356489 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.356493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.356665 (+ 172us) server_negotiation.cc:407] Connection header received
0927 15:02:48.356782 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.356785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.356830 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.356915 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.358237 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.358730 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.359815 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.360131 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.360237 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.361468 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:48.361471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:48.361478 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:48.361497 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:48.362960 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.363413 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.363416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.363417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.363572 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.363871 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.363873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.363874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.364021 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:48.364093 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.364388 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.364468 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:02:48.364784 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.356234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43961 (local address 127.0.98.3:38061)
0927 15:02:48.356577 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.356590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.356706 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.357837 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.357841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.358021 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:48.358119 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.358125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.358865 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.358870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.359701 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.359710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.360249 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.360253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.360371 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.360895 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:48.360913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:48.361348 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:48.363050 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.363054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.363057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.363316 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.363668 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.363671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.363676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.363772 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.364118 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:48.364124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:48.364234 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.364343 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.364501 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:02:48.386592 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:48.380064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.0.98.62:43961)
0927 15:02:48.380195 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.380199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.380211 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:48.380252 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.380254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.380293 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.380369 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.381038 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.381556 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.382503 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.382719 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.382813 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.386373 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53294: BlockingRecv error: recv got EOF from 127.0.0.1:53294 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:48.395736 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.388035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.0.98.62:43961)
0927 15:02:48.388185 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.388188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.388233 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:02:48.388333 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.388335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.388372 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.388439 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.389026 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.389535 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.390350 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.390555 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.390638 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.393034 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:48.393037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:48.393039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:48.393063 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:48.394358 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.394757 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.394760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.394761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.394834 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.395054 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.395056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.395057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.395226 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:48.395326 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.395519 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.395609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:02:48.398751 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:48.402189 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:48.416517 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.410768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.0.98.1:39649)
0927 15:02:48.411504 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.411511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.412073 (+ 562us) server_negotiation.cc:407] Connection header received
0927 15:02:48.412289 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.412294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.412448 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.412546 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.412854 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.413590 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.415345 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.415851 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.415991 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.416141 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.416306 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.416403 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":64,"threads_started":1}
I20260927 15:02:48.416512 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.410355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33505 (local address 127.0.0.1:39674)
0927 15:02:48.411040 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.411090 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.411231 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.412076 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.412078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.412093 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.412168 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.412174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.413665 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.413670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.414940 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.414951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.416024 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.416028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.416136 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.416158 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.416214 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.416332 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":140,"spinlock_wait_cycles":7424,"thread_start_us":102,"threads_started":1}
I20260927 15:02:48.416504 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.410615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39649 (local address 127.0.0.1:53676)
0927 15:02:48.411981 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.411994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.412140 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.412669 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.412671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.412684 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.412746 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.412750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.413723 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.413726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.415176 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.415183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.415977 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.415981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.416097 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.416133 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.416275 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.416349 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":88,"thread_start_us":97,"threads_started":1}
I20260927 15:02:48.416684 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.410209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41659 (local address 127.0.0.1:50478)
0927 15:02:48.411039 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.411096 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.411230 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.411775 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.411781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.411802 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.411947 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.411954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.413564 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.413567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.415149 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.415160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.416326 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.416329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.416410 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.416430 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.416494 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.416563 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":142,"spinlock_wait_cycles":10880,"thread_start_us":75,"threads_started":1}
I20260927 15:02:48.417018 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.411300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.0.98.2:33505)
0927 15:02:48.411653 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.411666 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.411689 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:02:48.411762 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.411768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.411935 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.412308 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.412426 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.413505 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.415071 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.415883 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.416574 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.416731 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.416784 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.416882 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":116,"threads_started":1}
I20260927 15:02:48.417162 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.410932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.98.3:41659)
0927 15:02:48.411262 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.411275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.411303 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:02:48.411376 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.411382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.411602 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.412110 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.412240 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.413442 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.415273 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.416022 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.416113 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.416270 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.416930 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.417021 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20260927 15:02:48.420059 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4bf6b2906b48a5acea9806b7557662 (DEFAULT_TABLE table=test-table [id=a0d3d913968944728f4af63c3c62c165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.420070 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4bf6b2906b48a5acea9806b7557662 (DEFAULT_TABLE table=test-table [id=a0d3d913968944728f4af63c3c62c165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.421237 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4bf6b2906b48a5acea9806b7557662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.421239 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4bf6b2906b48a5acea9806b7557662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.422873 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4bf6b2906b48a5acea9806b7557662 (DEFAULT_TABLE table=test-table [id=a0d3d913968944728f4af63c3c62c165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.423866 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4bf6b2906b48a5acea9806b7557662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.429090 8428 tablet_bootstrap.cc:492] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Bootstrap starting.
I20260927 15:02:48.430804 8430 tablet_bootstrap.cc:492] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Bootstrap starting.
I20260927 15:02:48.431463 8429 tablet_bootstrap.cc:492] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Bootstrap starting.
I20260927 15:02:48.431517 8428 tablet_bootstrap.cc:654] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.432404 8428 log.cc:824] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:48.432590 8430 tablet_bootstrap.cc:654] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.433349 8430 log.cc:824] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:48.433408 8429 tablet_bootstrap.cc:654] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.434221 8429 log.cc:824] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:48.434521 8428 tablet_bootstrap.cc:492] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: No bootstrap required, opened a new log
I20260927 15:02:48.434731 8428 ts_tablet_manager.cc:1397] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:02:48.435317 8430 tablet_bootstrap.cc:492] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: No bootstrap required, opened a new log
I20260927 15:02:48.435561 8430 ts_tablet_manager.cc:1397] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:02:48.436077 8429 tablet_bootstrap.cc:492] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: No bootstrap required, opened a new log
I20260927 15:02:48.436272 8429 ts_tablet_manager.cc:1397] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:02:48.439359 8429 raft_consensus.cc:348] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.439283 8430 raft_consensus.cc:348] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.439283 8428 raft_consensus.cc:348] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.439585 8430 raft_consensus.cc:374] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.439585 8429 raft_consensus.cc:374] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.439616 8428 raft_consensus.cc:374] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.439649 8429 raft_consensus.cc:729] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b2f430de574935bd3cd269226b726f, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.439668 8428 raft_consensus.cc:729] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c29d132e0439e87df1556a51e4e90, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.439684 8430 raft_consensus.cc:729] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17e138e781348d885de9934dde1a0fd, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.440124 8429 consensus_queue.cc:260] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.440155 8430 consensus_queue.cc:260] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.440155 8428 consensus_queue.cc:260] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.440317 8434 raft_consensus.cc:482] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:48.440490 8434 raft_consensus.cc:504] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.440872 8260 heartbeater.cc:503] Master 127.0.98.62:43961 was elected leader, sending a full tablet report...
I20260927 15:02:48.441154 8430 ts_tablet_manager.cc:1428] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:02:48.441751 8434 leader_election.cc:288] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659), c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505)
I20260927 15:02:48.441951 8400 heartbeater.cc:503] Master 127.0.98.62:43961 was elected leader, sending a full tablet report...
I20260927 15:02:48.442970 8428 ts_tablet_manager.cc:1428] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:02:48.443004 8429 ts_tablet_manager.cc:1428] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260927 15:02:48.443760 8120 heartbeater.cc:503] Master 127.0.98.62:43961 was elected leader, sending a full tablet report...
I20260927 15:02:48.447854 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.441934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41659 (local address 127.0.98.1:46943)
0927 15:02:48.443328 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.443345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.443489 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.443877 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.443880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.443900 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.444045 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.444051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.445127 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.445130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.446586 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.446593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.447402 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.447405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.447502 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.447526 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.447610 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.447697 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"thread_start_us":108,"threads_started":1}
I20260927 15:02:48.447881 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.442282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46943 (local address 127.0.98.3:41659)
0927 15:02:48.442385 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.442390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.443416 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 15:02:48.443631 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.443636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.443695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.443740 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.444186 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.444907 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.446723 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.447153 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.447213 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.447311 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.447669 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.447754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:02:48.448755 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf6b2906b48a5acea9806b7557662" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f" is_pre_election: true
I20260927 15:02:48.449043 8354 raft_consensus.cc:2454] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 0.
I20260927 15:02:48.449506 8007 leader_election.cc:302] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, d35c29d132e0439e87df1556a51e4e90; no voters:
I20260927 15:02:48.449753 8434 raft_consensus.cc:2781] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:48.449857 8434 raft_consensus.cc:482] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:48.449944 8434 raft_consensus.cc:3037] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.450305 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.443519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48611 (local address 127.0.98.2:33505)
0927 15:02:48.443629 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.443633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.446155 (+ 2522us) server_negotiation.cc:407] Connection header received
0927 15:02:48.446339 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.446342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.446412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.446455 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.446830 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.447551 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.449112 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.449752 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.449897 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.450032 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.450145 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.450216 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:02:48.450361 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.443465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33505 (local address 127.0.98.1:48611)
0927 15:02:48.446065 (+ 2600us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.446080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.446217 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.446571 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.446574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.446590 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.446704 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.446711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.447841 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.447846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.448995 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.449006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.449895 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.449901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.450011 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.450037 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.450126 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.450207 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2528,"thread_start_us":93,"threads_started":1}
I20260927 15:02:48.450917 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf6b2906b48a5acea9806b7557662" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17e138e781348d885de9934dde1a0fd" is_pre_election: true
I20260927 15:02:48.451200 8214 raft_consensus.cc:2454] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 0.
I20260927 15:02:48.451280 8434 raft_consensus.cc:504] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.451673 8434 leader_election.cc:288] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 election: Requested vote from peers 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659), c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505)
I20260927 15:02:48.452067 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf6b2906b48a5acea9806b7557662" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f"
I20260927 15:02:48.452195 8354 raft_consensus.cc:3037] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.453678 8354 raft_consensus.cc:2454] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 1.
I20260927 15:02:48.454015 8007 leader_election.cc:302] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, d35c29d132e0439e87df1556a51e4e90; no voters:
I20260927 15:02:48.454190 8434 raft_consensus.cc:2781] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:48.454437 8434 raft_consensus.cc:686] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 1 LEADER]: Becoming Leader. State: Replica: d35c29d132e0439e87df1556a51e4e90, State: Running, Role: LEADER
I20260927 15:02:48.454842 8434 consensus_queue.cc:237] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } }
I20260927 15:02:48.455483 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf6b2906b48a5acea9806b7557662" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17e138e781348d885de9934dde1a0fd"
I20260927 15:02:48.455614 8214 raft_consensus.cc:3037] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.456925 8214 raft_consensus.cc:2454] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 1.
I20260927 15:02:48.458582 7912 catalog_manager.cc:5495] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c29d132e0439e87df1556a51e4e90 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d35c29d132e0439e87df1556a51e4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } health_report { overall_health: HEALTHY } } }
I20260927 15:02:48.474906 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.471601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.0.98.1:39649)
0927 15:02:48.471745 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.471749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.471786 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:02:48.471852 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.471854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.471910 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.471990 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:48.472194 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.472680 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.473457 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.473682 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.473762 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.474052 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.474475 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.474559 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.474690 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.474736 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:02:48.480455 8354 raft_consensus.cc:1261] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Refusing update from remote peer d35c29d132e0439e87df1556a51e4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:48.480454 8214 raft_consensus.cc:1261] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Refusing update from remote peer d35c29d132e0439e87df1556a51e4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:48.481115 8439 consensus_queue.cc:1035] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:48.481284 8434 consensus_queue.cc:1035] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:48.489979 8443 mvcc.cc:205] Tried to move back new op lower bound from 7333975525289431040 to 7333975525195108352. Current Snapshot: MvccSnapshot[applied={T|T < 7333975525289431040}]
I20260927 15:02:48.497956 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:48.498153 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:48.500547 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Sending DeleteTablet for 3 replicas of tablet 4e4bf6b2906b48a5acea9806b7557662
I20260927 15:02:48.501356 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e4bf6b2906b48a5acea9806b7557662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53676
I20260927 15:02:48.501361 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e4bf6b2906b48a5acea9806b7557662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50478
I20260927 15:02:48.501400 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e4bf6b2906b48a5acea9806b7557662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39674
I20260927 15:02:48.501904 8453 tablet_replica.cc:330] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: stopping tablet replica
I20260927 15:02:48.502224 8453 raft_consensus.cc:2229] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:48.502597 8453 raft_consensus.cc:2258] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:48.502698 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:02:48.503072 8454 tablet_replica.cc:330] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: stopping tablet replica
W20260927 15:02:48.503125 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:48.503356 8454 raft_consensus.cc:2229] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:48.503728 8454 raft_consensus.cc:2258] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:48.504156 8452 tablet_replica.cc:330] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: stopping tablet replica
I20260927 15:02:48.504415 8452 raft_consensus.cc:2229] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:48.504747 8452 raft_consensus.cc:2258] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:48.506119 8453 ts_tablet_manager.cc:1905] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:48.506860 8454 ts_tablet_manager.cc:1905] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:48.508371 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 5a08c47eef19439e956115c209f1c2e4 (DEFAULT_TABLE table=test-table [id=ba2b82380b9f4ac18b1c10b9e9599937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.508641 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a08c47eef19439e956115c209f1c2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.509341 8453 ts_tablet_manager.cc:1918] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:48.509505 8453 log.cc:1196] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/4e4bf6b2906b48a5acea9806b7557662
I20260927 15:02:48.509505 8454 ts_tablet_manager.cc:1918] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:48.509580 8454 log.cc:1196] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/4e4bf6b2906b48a5acea9806b7557662
I20260927 15:02:48.509941 8454 ts_tablet_manager.cc:1939] T 4e4bf6b2906b48a5acea9806b7557662 P d35c29d132e0439e87df1556a51e4e90: Deleting consensus metadata
I20260927 15:02:48.509975 8453 ts_tablet_manager.cc:1939] T 4e4bf6b2906b48a5acea9806b7557662 P 36b2f430de574935bd3cd269226b726f: Deleting consensus metadata
I20260927 15:02:48.510269 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 5a08c47eef19439e956115c209f1c2e4 (DEFAULT_TABLE table=test-table [id=ba2b82380b9f4ac18b1c10b9e9599937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.510502 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a08c47eef19439e956115c209f1c2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.511091 7902 catalog_manager.cc:4841] TS d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649): tablet 4e4bf6b2906b48a5acea9806b7557662 (table test-table [id=a0d3d913968944728f4af63c3c62c165]) successfully deleted
I20260927 15:02:48.511807 8452 ts_tablet_manager.cc:1905] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:48.512518 8428 tablet_bootstrap.cc:492] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Bootstrap starting.
I20260927 15:02:48.512816 7903 catalog_manager.cc:4841] TS 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659): tablet 4e4bf6b2906b48a5acea9806b7557662 (table test-table [id=a0d3d913968944728f4af63c3c62c165]) successfully deleted
I20260927 15:02:48.513477 8428 tablet_bootstrap.cc:654] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.514256 8430 tablet_bootstrap.cc:492] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: Bootstrap starting.
I20260927 15:02:48.514577 8452 ts_tablet_manager.cc:1918] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:48.514671 8452 log.cc:1196] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/4e4bf6b2906b48a5acea9806b7557662
I20260927 15:02:48.514710 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 5a08c47eef19439e956115c209f1c2e4 (DEFAULT_TABLE table=test-table [id=ba2b82380b9f4ac18b1c10b9e9599937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.514962 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a08c47eef19439e956115c209f1c2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.514981 8452 ts_tablet_manager.cc:1939] T 4e4bf6b2906b48a5acea9806b7557662 P c17e138e781348d885de9934dde1a0fd: Deleting consensus metadata
I20260927 15:02:48.515208 8430 tablet_bootstrap.cc:654] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.516069 7901 catalog_manager.cc:4841] TS c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505): tablet 4e4bf6b2906b48a5acea9806b7557662 (table test-table [id=a0d3d913968944728f4af63c3c62c165]) successfully deleted
I20260927 15:02:48.516270 8430 tablet_bootstrap.cc:492] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: No bootstrap required, opened a new log
I20260927 15:02:48.516336 8430 ts_tablet_manager.cc:1397] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:48.516723 8430 raft_consensus.cc:348] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.516819 8430 raft_consensus.cc:374] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.516847 8430 raft_consensus.cc:729] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17e138e781348d885de9934dde1a0fd, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.516954 8428 tablet_bootstrap.cc:492] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: No bootstrap required, opened a new log
I20260927 15:02:48.516965 8430 consensus_queue.cc:260] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.517028 8428 ts_tablet_manager.cc:1397] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:02:48.517223 8430 ts_tablet_manager.cc:1428] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:02:48.517378 8429 tablet_bootstrap.cc:492] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Bootstrap starting.
I20260927 15:02:48.517504 8428 raft_consensus.cc:348] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.517601 8428 raft_consensus.cc:374] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.517632 8428 raft_consensus.cc:729] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c29d132e0439e87df1556a51e4e90, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.517741 8428 consensus_queue.cc:260] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.518021 8428 ts_tablet_manager.cc:1428] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:48.518321 8429 tablet_bootstrap.cc:654] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.519513 8429 tablet_bootstrap.cc:492] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: No bootstrap required, opened a new log
I20260927 15:02:48.519615 8429 ts_tablet_manager.cc:1397] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:48.520062 8429 raft_consensus.cc:348] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.520171 8429 raft_consensus.cc:374] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.520206 8429 raft_consensus.cc:729] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b2f430de574935bd3cd269226b726f, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.520318 8429 consensus_queue.cc:260] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.520605 8429 ts_tablet_manager.cc:1428] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:48.528357 8435 raft_consensus.cc:482] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:48.528484 8435 raft_consensus.cc:504] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.529656 8435 leader_election.cc:288] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649), 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659)
I20260927 15:02:48.534255 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.529930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35733 (local address 127.0.98.3:41659)
0927 15:02:48.530060 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.530064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.530346 (+ 282us) server_negotiation.cc:407] Connection header received
0927 15:02:48.530480 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.530483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.530525 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.530607 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.530859 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.531341 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.533045 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.533644 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.533754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.533844 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.534068 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.534138 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:02:48.534286 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.529914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41659 (local address 127.0.98.2:35733)
0927 15:02:48.530251 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.530263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.530383 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.530620 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.530623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.530638 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.530757 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.530762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.531448 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.531453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.532907 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.532917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.533815 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.533819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.533938 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.533964 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.534038 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.534120 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":41,"thread_start_us":90,"threads_started":1}
I20260927 15:02:48.534427 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.529856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39649 (local address 127.0.98.2:43447)
0927 15:02:48.529973 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.529991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.530084 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.530492 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.530495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.530512 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.530619 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.530625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.531620 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.531623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.533118 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.533126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.534081 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.534084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.534176 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.534195 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.534260 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.534328 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":259}
I20260927 15:02:48.534909 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.530109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43447 (local address 127.0.98.1:39649)
0927 15:02:48.530219 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.530223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.530235 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:48.530279 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.530281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.530322 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.530390 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.530738 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.531508 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.533246 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.533864 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.533958 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.534028 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.534725 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.534783 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 15:02:48.534905 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a08c47eef19439e956115c209f1c2e4" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f" is_pre_election: true
I20260927 15:02:48.535040 8354 raft_consensus.cc:2454] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 0.
I20260927 15:02:48.535359 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a08c47eef19439e956115c209f1c2e4" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c29d132e0439e87df1556a51e4e90" is_pre_election: true
I20260927 15:02:48.535604 8058 raft_consensus.cc:2454] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 0.
I20260927 15:02:48.535982 8146 leader_election.cc:302] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c17e138e781348d885de9934dde1a0fd, d35c29d132e0439e87df1556a51e4e90; no voters:
I20260927 15:02:48.536258 8435 raft_consensus.cc:2781] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:48.536312 8435 raft_consensus.cc:482] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:48.536348 8435 raft_consensus.cc:3037] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.537528 8435 raft_consensus.cc:504] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.537951 8435 leader_election.cc:288] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 election: Requested vote from peers d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649), 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659)
I20260927 15:02:48.538316 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a08c47eef19439e956115c209f1c2e4" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c29d132e0439e87df1556a51e4e90"
I20260927 15:02:48.538439 8058 raft_consensus.cc:3037] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.538439 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a08c47eef19439e956115c209f1c2e4" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f"
I20260927 15:02:48.538542 8354 raft_consensus.cc:3037] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.539371 8354 raft_consensus.cc:2454] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 1.
I20260927 15:02:48.539613 8058 raft_consensus.cc:2454] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 1.
I20260927 15:02:48.539695 8147 leader_election.cc:302] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, c17e138e781348d885de9934dde1a0fd; no voters:
I20260927 15:02:48.539881 8435 raft_consensus.cc:2781] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:48.540132 8435 raft_consensus.cc:686] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 1 LEADER]: Becoming Leader. State: Replica: c17e138e781348d885de9934dde1a0fd, State: Running, Role: LEADER
I20260927 15:02:48.540392 8435 consensus_queue.cc:237] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.542908 7916 catalog_manager.cc:5495] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c17e138e781348d885de9934dde1a0fd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c17e138e781348d885de9934dde1a0fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521367, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:48.584296 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.575655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43961 (local address 127.0.0.1:53316)
0927 15:02:48.575946 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.575964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.576069 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.576318 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.576321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.576728 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:48.576821 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.576827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.577863 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.577866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.578403 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.578409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.579001 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.579004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.579062 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.579651 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:48.579662 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:48.580994 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:48.582844 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.582852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.582857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.583069 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.583342 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:48.583344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:48.583345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:48.583418 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:48.583807 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:48.583812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:48.583910 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.584054 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.584165 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":127,"threads_started":1}
I20260927 15:02:48.584420 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.575700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.0.98.62:43961)
0927 15:02:48.575834 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.575839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.576043 (+ 204us) server_negotiation.cc:407] Connection header received
0927 15:02:48.576144 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.576147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.576193 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.576292 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.576935 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.577740 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.578525 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.578879 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.579002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.581135 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:48.581139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:48.581142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:48.581167 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:48.582731 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.583166 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.583170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.583172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.583255 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.583501 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.583504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.583506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.583710 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:48.583805 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.584131 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.584242 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:02:48.588536 7916 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53316:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:02:48.589042 8121 tablet.cc:2369] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:02:48.589402 7916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:48.594223 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 19713fd5298f4099ad0378dc407681e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=909aa23de17148b7b6cf3d016041fd3d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:48.594223 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 19713fd5298f4099ad0378dc407681e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=909aa23de17148b7b6cf3d016041fd3d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:48.594585 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19713fd5298f4099ad0378dc407681e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.594585 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19713fd5298f4099ad0378dc407681e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.597223 8428 tablet_bootstrap.cc:492] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: Bootstrap starting.
I20260927 15:02:48.597249 8430 tablet_bootstrap.cc:492] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: Bootstrap starting.
I20260927 15:02:48.597572 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 19713fd5298f4099ad0378dc407681e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=909aa23de17148b7b6cf3d016041fd3d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:02:48.597950 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19713fd5298f4099ad0378dc407681e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.598130 8428 tablet_bootstrap.cc:654] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.598174 8430 tablet_bootstrap.cc:654] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.599538 8428 tablet_bootstrap.cc:492] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: No bootstrap required, opened a new log
I20260927 15:02:48.599619 8428 ts_tablet_manager.cc:1397] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:48.600178 8428 raft_consensus.cc:348] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.600281 8428 raft_consensus.cc:374] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.600307 8428 raft_consensus.cc:729] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c29d132e0439e87df1556a51e4e90, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.600448 8428 consensus_queue.cc:260] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.600669 8429 tablet_bootstrap.cc:492] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: Bootstrap starting.
I20260927 15:02:48.600688 8428 ts_tablet_manager.cc:1428] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 15:02:48.600993 8401 tablet.cc:2369] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:48.601688 8429 tablet_bootstrap.cc:654] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.601737 8430 tablet_bootstrap.cc:492] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: No bootstrap required, opened a new log
I20260927 15:02:48.601827 8430 ts_tablet_manager.cc:1397] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:02:48.602111 8448 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.602250 8448 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:48.602272 8430 raft_consensus.cc:348] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.602375 8430 raft_consensus.cc:374] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.602397 8430 raft_consensus.cc:729] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17e138e781348d885de9934dde1a0fd, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.602522 8430 consensus_queue.cc:260] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.602770 8435 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.602871 8430 ts_tablet_manager.cc:1428] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 15:02:48.603299 8261 tablet.cc:2369] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:02:48.602880 8435 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:48.606556 8429 tablet_bootstrap.cc:492] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: No bootstrap required, opened a new log
I20260927 15:02:48.606662 8429 ts_tablet_manager.cc:1397] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:02:48.607149 8429 raft_consensus.cc:348] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.607275 8429 raft_consensus.cc:374] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.607314 8429 raft_consensus.cc:729] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b2f430de574935bd3cd269226b726f, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.607460 8429 consensus_queue.cc:260] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.607758 8436 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.607914 8436 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:48.608201 8429 ts_tablet_manager.cc:1428] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 15:02:48.730799 8436 raft_consensus.cc:482] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:48.730950 8436 raft_consensus.cc:504] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.732002 8436 leader_election.cc:288] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649), c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505)
I20260927 15:02:48.735707 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.732213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39649 (local address 127.0.98.3:56789)
0927 15:02:48.732517 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.732534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.732633 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.732884 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.732887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.732903 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.733018 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.733024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.733797 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.733803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.734596 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.734605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.735250 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.735253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.735357 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.735383 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.735489 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.735569 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260927 15:02:48.735707 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.732396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56789 (local address 127.0.98.1:39649)
0927 15:02:48.732614 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.732618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.732669 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:02:48.732733 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.732735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.732784 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.732876 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.733131 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.733653 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.734728 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.735127 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.735250 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.735316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.735505 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.735583 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260927 15:02:48.736266 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.732241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33505 (local address 127.0.98.3:41395)
0927 15:02:48.732648 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.732659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.732783 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.733020 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.733022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.733037 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:02:48.733515 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.733522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.734269 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.734275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.735039 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.735047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.735688 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.735692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.735945 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.735980 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.736052 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.736131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":134,"threads_started":1}
I20260927 15:02:48.736263 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.732371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41395 (local address 127.0.98.2:33505)
0927 15:02:48.732529 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.732534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.732742 (+ 208us) server_negotiation.cc:407] Connection header received
0927 15:02:48.732874 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.732877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.732925 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.733010 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:02:48.733656 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.734151 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.735180 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.735559 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.735685 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.735743 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.736091 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.736160 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:02:48.736251 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19713fd5298f4099ad0378dc407681e4" candidate_uuid: "36b2f430de574935bd3cd269226b726f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c29d132e0439e87df1556a51e4e90" is_pre_election: true
I20260927 15:02:48.736419 8058 raft_consensus.cc:2454] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b2f430de574935bd3cd269226b726f in term 0.
I20260927 15:02:48.736739 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19713fd5298f4099ad0378dc407681e4" candidate_uuid: "36b2f430de574935bd3cd269226b726f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17e138e781348d885de9934dde1a0fd" is_pre_election: true
I20260927 15:02:48.736868 8214 raft_consensus.cc:2454] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b2f430de574935bd3cd269226b726f in term 0.
I20260927 15:02:48.736858 8286 leader_election.cc:302] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, d35c29d132e0439e87df1556a51e4e90; no voters:
I20260927 15:02:48.737186 8436 raft_consensus.cc:2781] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:48.737267 8436 raft_consensus.cc:482] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:48.737339 8436 raft_consensus.cc:3037] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.738435 8436 raft_consensus.cc:504] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.738814 8436 leader_election.cc:288] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [CANDIDATE]: Term 1 election: Requested vote from peers d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649), c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505)
I20260927 15:02:48.739157 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19713fd5298f4099ad0378dc407681e4" candidate_uuid: "36b2f430de574935bd3cd269226b726f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c29d132e0439e87df1556a51e4e90"
I20260927 15:02:48.739197 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19713fd5298f4099ad0378dc407681e4" candidate_uuid: "36b2f430de574935bd3cd269226b726f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17e138e781348d885de9934dde1a0fd"
I20260927 15:02:48.739293 8058 raft_consensus.cc:3037] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.739317 8214 raft_consensus.cc:3037] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:48.740195 8214 raft_consensus.cc:2454] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b2f430de574935bd3cd269226b726f in term 1.
I20260927 15:02:48.740219 8058 raft_consensus.cc:2454] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b2f430de574935bd3cd269226b726f in term 1.
I20260927 15:02:48.740523 8285 leader_election.cc:302] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, c17e138e781348d885de9934dde1a0fd; no voters:
I20260927 15:02:48.740691 8436 raft_consensus.cc:2781] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:48.740907 8436 raft_consensus.cc:686] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [term 1 LEADER]: Becoming Leader. State: Replica: 36b2f430de574935bd3cd269226b726f, State: Running, Role: LEADER
I20260927 15:02:48.741165 8436 consensus_queue.cc:237] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.742388 8471 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: TxnStatusTablet state changed. Reason: New leader 36b2f430de574935bd3cd269226b726f. Latest consensus state: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.742507 8471 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:48.742779 8473 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:02:48.742945 8473 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:02:48.743639 7916 catalog_manager.cc:5495] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36b2f430de574935bd3cd269226b726f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } health_report { overall_health: HEALTHY } } }
I20260927 15:02:48.757447 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.753405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41659 (local address 127.0.0.1:50482)
0927 15:02:48.753568 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.753584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.753680 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.754045 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.754047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.754056 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:48.754155 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.754159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.755206 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.755208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.755806 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.755813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.756352 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.756355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.756415 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.756457 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.757205 (+ 748us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.757217 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.757272 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.757341 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 15:02:48.757467 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.753497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.0.98.3:41659)
0927 15:02:48.753636 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.753641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.753657 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:02:48.753841 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.753844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.753922 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.754030 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:48.754260 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.755099 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.755897 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.756249 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.756353 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.756596 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.757102 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.757226 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.757284 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.757336 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:02:48.760190 8214 raft_consensus.cc:1261] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Refusing update from remote peer 36b2f430de574935bd3cd269226b726f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:48.760385 8058 raft_consensus.cc:1261] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Refusing update from remote peer 36b2f430de574935bd3cd269226b726f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:48.760812 8436 consensus_queue.cc:1035] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:48.760972 8471 consensus_queue.cc:1035] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:48.762547 8435 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: TxnStatusTablet state changed. Reason: New leader 36b2f430de574935bd3cd269226b726f. Latest consensus state: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.762652 8435 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:48.764444 8448 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: TxnStatusTablet state changed. Reason: New leader 36b2f430de574935bd3cd269226b726f. Latest consensus state: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.764562 8448 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:48.765611 8471 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.765702 8471 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:48.765583 8436 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.765823 8436 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P 36b2f430de574935bd3cd269226b726f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:02:48.766919 8448 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.767052 8448 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P d35c29d132e0439e87df1556a51e4e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:48.767197 8435 tablet_replica.cc:439] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b2f430de574935bd3cd269226b726f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } } }
I20260927 15:02:48.767297 8435 tablet_replica.cc:442] T 19713fd5298f4099ad0378dc407681e4 P c17e138e781348d885de9934dde1a0fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:02:48.767982 8444 mvcc.cc:205] Tried to move back new op lower bound from 7333975526437347328 to 7333975526366478336. Current Snapshot: MvccSnapshot[applied={T|T < 7333975526437347328 or (T in {7333975526437347328})}]
I20260927 15:02:48.768183 8442 mvcc.cc:205] Tried to move back new op lower bound from 7333975526437347328 to 7333975526366478336. Current Snapshot: MvccSnapshot[applied={T|T < 7333975526437347328 or (T in {7333975526437347328})}]
I20260927 15:02:48.777452 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.773542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.0.98.2:33505)
0927 15:02:48.773659 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.773663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.773677 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:48.773851 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.773854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.773918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.774025 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:48.774247 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.775017 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.775846 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.776216 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.776318 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.776541 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.777084 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.777215 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.777247 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.777300 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:02:48.788053 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.784286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41659 (local address 127.0.98.2:49251)
0927 15:02:48.784583 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.784596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.784678 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.784921 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.784923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.784928 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:48.785034 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.785038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.786026 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.786028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.786799 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.786807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.787293 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.787297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.787392 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.787430 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.787776 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.787784 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.787848 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.787916 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
I20260927 15:02:48.788039 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.784392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49251 (local address 127.0.98.3:41659)
0927 15:02:48.784514 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.784519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.784669 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:02:48.784772 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.784775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.784825 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.784909 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:48.785107 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.785933 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.786940 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.787171 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.787297 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.787563 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.787685 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.787777 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.787859 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.787902 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:02:48.794704 8354 raft_consensus.cc:1261] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Refusing update from remote peer c17e138e781348d885de9934dde1a0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:48.794835 8058 raft_consensus.cc:1261] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Refusing update from remote peer c17e138e781348d885de9934dde1a0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:48.795225 8459 consensus_queue.cc:1035] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:48.795468 8435 consensus_queue.cc:1035] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:48.818897 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.815197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33505 (local address 127.0.98.3:54031)
0927 15:02:48.815369 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:48.815384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:48.815467 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:48.815720 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:48.815723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:48.815731 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:48.815849 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.815855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.816680 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.816687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.817599 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:48.817609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.818276 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.818279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:48.818350 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.818392 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.818676 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.818680 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:02:48.818731 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 15:02:48.818792 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 15:02:48.818923 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.815302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54031 (local address 127.0.98.2:33505)
0927 15:02:48.815412 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.815416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.815428 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:48.815567 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.815569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.815623 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.815685 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:48.815947 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.816566 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.817812 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.818168 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.818272 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.818515 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:48.818595 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:48.818667 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.818747 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.818796 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:02:48.846756 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:48.846925 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:48.848984 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Sending DeleteTablet for 3 replicas of tablet 5a08c47eef19439e956115c209f1c2e4
I20260927 15:02:48.849654 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a08c47eef19439e956115c209f1c2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53676
I20260927 15:02:48.849880 8454 tablet_replica.cc:330] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: stopping tablet replica
I20260927 15:02:48.849957 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a08c47eef19439e956115c209f1c2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39674
I20260927 15:02:48.850039 8454 raft_consensus.cc:2229] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:48.850103 8452 tablet_replica.cc:330] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: stopping tablet replica
I20260927 15:02:48.850183 8454 raft_consensus.cc:2258] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:48.850216 8452 raft_consensus.cc:2229] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:48.850471 8452 raft_consensus.cc:2258] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:48.851033 8454 ts_tablet_manager.cc:1905] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:48.851107 8452 ts_tablet_manager.cc:1905] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:48.853699 8452 ts_tablet_manager.cc:1918] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:48.853785 8452 log.cc:1196] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/5a08c47eef19439e956115c209f1c2e4
I20260927 15:02:48.854020 8452 ts_tablet_manager.cc:1939] T 5a08c47eef19439e956115c209f1c2e4 P c17e138e781348d885de9934dde1a0fd: Deleting consensus metadata
I20260927 15:02:48.854278 8454 ts_tablet_manager.cc:1918] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:48.854414 8454 log.cc:1196] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/5a08c47eef19439e956115c209f1c2e4
I20260927 15:02:48.854439 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a08c47eef19439e956115c209f1c2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50478
I20260927 15:02:48.854643 8453 tablet_replica.cc:330] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: stopping tablet replica
I20260927 15:02:48.854758 8453 raft_consensus.cc:2229] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:48.854877 8453 raft_consensus.cc:2258] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:48.855286 8454 ts_tablet_manager.cc:1939] T 5a08c47eef19439e956115c209f1c2e4 P d35c29d132e0439e87df1556a51e4e90: Deleting consensus metadata
I20260927 15:02:48.855705 8453 ts_tablet_manager.cc:1905] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:48.856077 7902 catalog_manager.cc:4841] TS d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649): tablet 5a08c47eef19439e956115c209f1c2e4 (table test-table [id=ba2b82380b9f4ac18b1c10b9e9599937]) successfully deleted
I20260927 15:02:48.858352 8453 ts_tablet_manager.cc:1918] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:48.858433 8453 log.cc:1196] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/5a08c47eef19439e956115c209f1c2e4
I20260927 15:02:48.858672 8453 ts_tablet_manager.cc:1939] T 5a08c47eef19439e956115c209f1c2e4 P 36b2f430de574935bd3cd269226b726f: Deleting consensus metadata
I20260927 15:02:48.862888 7903 catalog_manager.cc:4841] TS 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659): tablet 5a08c47eef19439e956115c209f1c2e4 (table test-table [id=ba2b82380b9f4ac18b1c10b9e9599937]) successfully deleted
I20260927 15:02:48.863862 7901 catalog_manager.cc:4841] TS c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505): tablet 5a08c47eef19439e956115c209f1c2e4 (table test-table [id=ba2b82380b9f4ac18b1c10b9e9599937]) successfully deleted
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:02:48.885849 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:48.879432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.0.98.62:43961)
0927 15:02:48.879563 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.879568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.879814 (+ 246us) server_negotiation.cc:407] Connection header received
0927 15:02:48.879979 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.879983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.880035 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.880133 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.880843 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.881370 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.882343 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.882564 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.882677 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.885660 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53322: BlockingRecv error: recv got EOF from 127.0.0.1:53322 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 15:02:48 dist-test-slave-ml7t krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 15:02:48.898589 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:48.890258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.0.98.62:43961)
0927 15:02:48.890372 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:48.890376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:48.890534 (+ 158us) server_negotiation.cc:407] Connection header received
0927 15:02:48.890724 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:48.890727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:48.890773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:48.890860 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:48.891443 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.891956 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.892776 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:48.892987 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:48.893105 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:48.895463 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:48.895467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:48.895470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:48.895503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:48.896896 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.897326 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.897331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.897333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.897410 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:48.897686 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:48.897690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:48.897691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:48.897918 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:48.898047 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:48.898300 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:02:48.898419 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:02:48.901392 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:48.901898 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:48.906946 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 29909988929d40b0907ba97f4d5f490b (DEFAULT_TABLE table=test-table [id=13c53e458bfa43928bcb7db000db3662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.906945 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 29909988929d40b0907ba97f4d5f490b (DEFAULT_TABLE table=test-table [id=13c53e458bfa43928bcb7db000db3662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.907244 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29909988929d40b0907ba97f4d5f490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.907258 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29909988929d40b0907ba97f4d5f490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.909569 8430 tablet_bootstrap.cc:492] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: Bootstrap starting.
I20260927 15:02:48.909569 8428 tablet_bootstrap.cc:492] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: Bootstrap starting.
I20260927 15:02:48.910552 8430 tablet_bootstrap.cc:654] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.910614 8428 tablet_bootstrap.cc:654] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.911475 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 29909988929d40b0907ba97f4d5f490b (DEFAULT_TABLE table=test-table [id=13c53e458bfa43928bcb7db000db3662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:48.911746 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29909988929d40b0907ba97f4d5f490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:48.913333 8430 tablet_bootstrap.cc:492] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: No bootstrap required, opened a new log
I20260927 15:02:48.913419 8430 ts_tablet_manager.cc:1397] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 15:02:48.913918 8430 raft_consensus.cc:348] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.914036 8430 raft_consensus.cc:374] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.914067 8430 raft_consensus.cc:729] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17e138e781348d885de9934dde1a0fd, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.914181 8430 consensus_queue.cc:260] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.914317 8429 tablet_bootstrap.cc:492] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: Bootstrap starting.
I20260927 15:02:48.914451 8430 ts_tablet_manager.cc:1428] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:02:48.915244 8429 tablet_bootstrap.cc:654] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:48.917037 8429 tablet_bootstrap.cc:492] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: No bootstrap required, opened a new log
I20260927 15:02:48.917124 8429 ts_tablet_manager.cc:1397] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:48.917529 8428 tablet_bootstrap.cc:492] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: No bootstrap required, opened a new log
I20260927 15:02:48.917616 8428 ts_tablet_manager.cc:1397] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 15:02:48.917573 8429 raft_consensus.cc:348] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.917680 8429 raft_consensus.cc:374] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.917716 8429 raft_consensus.cc:729] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b2f430de574935bd3cd269226b726f, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.918147 8428 raft_consensus.cc:348] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.918263 8428 raft_consensus.cc:374] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:48.918228 8429 consensus_queue.cc:260] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.918295 8428 raft_consensus.cc:729] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c29d132e0439e87df1556a51e4e90, State: Initialized, Role: FOLLOWER
I20260927 15:02:48.918406 8428 consensus_queue.cc:260] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:48.918498 8429 ts_tablet_manager.cc:1428] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:48.918670 8428 ts_tablet_manager.cc:1428] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:49.042268 8435 raft_consensus.cc:482] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:49.042397 8435 raft_consensus.cc:504] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.042865 8435 leader_election.cc:288] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649), 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659)
I20260927 15:02:49.043334 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29909988929d40b0907ba97f4d5f490b" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f" is_pre_election: true
I20260927 15:02:49.043334 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29909988929d40b0907ba97f4d5f490b" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c29d132e0439e87df1556a51e4e90" is_pre_election: true
I20260927 15:02:49.043507 8354 raft_consensus.cc:2454] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 0.
I20260927 15:02:49.043507 8058 raft_consensus.cc:2454] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 0.
I20260927 15:02:49.043851 8147 leader_election.cc:302] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, c17e138e781348d885de9934dde1a0fd; no voters:
I20260927 15:02:49.044075 8435 raft_consensus.cc:2781] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:49.044133 8435 raft_consensus.cc:482] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:49.044162 8435 raft_consensus.cc:3037] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:49.045337 8435 raft_consensus.cc:504] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.045692 8435 leader_election.cc:288] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 election: Requested vote from peers d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649), 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659)
I20260927 15:02:49.046074 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29909988929d40b0907ba97f4d5f490b" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f"
I20260927 15:02:49.046125 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29909988929d40b0907ba97f4d5f490b" candidate_uuid: "c17e138e781348d885de9934dde1a0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c29d132e0439e87df1556a51e4e90"
I20260927 15:02:49.046182 8354 raft_consensus.cc:3037] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:49.046258 8058 raft_consensus.cc:3037] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:49.047176 8058 raft_consensus.cc:2454] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 1.
I20260927 15:02:49.047178 8354 raft_consensus.cc:2454] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c17e138e781348d885de9934dde1a0fd in term 1.
I20260927 15:02:49.047459 8147 leader_election.cc:302] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, c17e138e781348d885de9934dde1a0fd; no voters:
I20260927 15:02:49.047648 8435 raft_consensus.cc:2781] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:49.047757 8435 raft_consensus.cc:686] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 1 LEADER]: Becoming Leader. State: Replica: c17e138e781348d885de9934dde1a0fd, State: Running, Role: LEADER
I20260927 15:02:49.047921 8435 consensus_queue.cc:237] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.049646 7916 catalog_manager.cc:5495] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c17e138e781348d885de9934dde1a0fd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c17e138e781348d885de9934dde1a0fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } health_report { overall_health: UNKNOWN } } }
I20260927 15:02:49.071990 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:49.068696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.0.98.2:33505)
0927 15:02:49.068811 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:49.068815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:49.068927 (+ 112us) server_negotiation.cc:407] Connection header received
0927 15:02:49.069078 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:49.069080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:49.069123 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:49.069205 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:49.069451 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.070021 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.070890 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.071116 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.071229 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:49.071540 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:49.071612 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:49.071693 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:49.071784 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 15:02:49.071835 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 15:02:49.074015 8354 raft_consensus.cc:1261] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Refusing update from remote peer c17e138e781348d885de9934dde1a0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:49.074057 8058 raft_consensus.cc:1261] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Refusing update from remote peer c17e138e781348d885de9934dde1a0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:49.074487 8459 consensus_queue.cc:1035] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:49.074657 8435 consensus_queue.cc:1035] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:49.081494 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:49.081657 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:49.084266 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Sending DeleteTablet for 3 replicas of tablet 29909988929d40b0907ba97f4d5f490b
I20260927 15:02:49.084757 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet 29909988929d40b0907ba97f4d5f490b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53676
I20260927 15:02:49.084932 8454 tablet_replica.cc:330] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: stopping tablet replica
I20260927 15:02:49.084914 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet 29909988929d40b0907ba97f4d5f490b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39674
I20260927 15:02:49.085058 8454 raft_consensus.cc:2229] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:49.085048 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 29909988929d40b0907ba97f4d5f490b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50478
I20260927 15:02:49.085224 8454 raft_consensus.cc:2258] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:49.085266 8452 tablet_replica.cc:330] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: stopping tablet replica
I20260927 15:02:49.085407 8452 raft_consensus.cc:2229] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:49.085611 8452 raft_consensus.cc:2258] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:49.085829 8453 tablet_replica.cc:330] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: stopping tablet replica
I20260927 15:02:49.085949 8454 ts_tablet_manager.cc:1905] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:49.085932 8453 raft_consensus.cc:2229] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:49.086081 8453 raft_consensus.cc:2258] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:49.086234 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:02:49.086661 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:02:49.086863 8452 ts_tablet_manager.cc:1905] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:49.087109 8453 ts_tablet_manager.cc:1905] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:49.088511 8454 ts_tablet_manager.cc:1918] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:49.088579 8454 log.cc:1196] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/29909988929d40b0907ba97f4d5f490b
I20260927 15:02:49.088831 8454 ts_tablet_manager.cc:1939] T 29909988929d40b0907ba97f4d5f490b P d35c29d132e0439e87df1556a51e4e90: Deleting consensus metadata
I20260927 15:02:49.089478 8452 ts_tablet_manager.cc:1918] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:49.089488 8453 ts_tablet_manager.cc:1918] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:02:49.089576 8453 log.cc:1196] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/29909988929d40b0907ba97f4d5f490b
I20260927 15:02:49.089574 8452 log.cc:1196] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/29909988929d40b0907ba97f4d5f490b
I20260927 15:02:49.089870 8453 ts_tablet_manager.cc:1939] T 29909988929d40b0907ba97f4d5f490b P 36b2f430de574935bd3cd269226b726f: Deleting consensus metadata
I20260927 15:02:49.089898 8452 ts_tablet_manager.cc:1939] T 29909988929d40b0907ba97f4d5f490b P c17e138e781348d885de9934dde1a0fd: Deleting consensus metadata
I20260927 15:02:49.090657 7901 catalog_manager.cc:4841] TS c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505): tablet 29909988929d40b0907ba97f4d5f490b (table test-table [id=13c53e458bfa43928bcb7db000db3662]) successfully deleted
I20260927 15:02:49.090703 7903 catalog_manager.cc:4841] TS 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659): tablet 29909988929d40b0907ba97f4d5f490b (table test-table [id=13c53e458bfa43928bcb7db000db3662]) successfully deleted
I20260927 15:02:49.091477 8054 tablet_service.cc:1467] Processing CreateTablet for tablet e063129b5ed2412ab5eadfa1916afd1f (DEFAULT_TABLE table=test-table [id=367cb1dff25948a79577c594f1cd35c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:49.091734 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e063129b5ed2412ab5eadfa1916afd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:49.091713 8194 tablet_service.cc:1467] Processing CreateTablet for tablet e063129b5ed2412ab5eadfa1916afd1f (DEFAULT_TABLE table=test-table [id=367cb1dff25948a79577c594f1cd35c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:49.091825 8334 tablet_service.cc:1467] Processing CreateTablet for tablet e063129b5ed2412ab5eadfa1916afd1f (DEFAULT_TABLE table=test-table [id=367cb1dff25948a79577c594f1cd35c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:02:49.091924 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e063129b5ed2412ab5eadfa1916afd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:49.092084 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e063129b5ed2412ab5eadfa1916afd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:49.093707 8429 tablet_bootstrap.cc:492] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: Bootstrap starting.
I20260927 15:02:49.094018 8428 tablet_bootstrap.cc:492] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90: Bootstrap starting.
I20260927 15:02:49.094039 8430 tablet_bootstrap.cc:492] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: Bootstrap starting.
I20260927 15:02:49.094491 8429 tablet_bootstrap.cc:654] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:49.094762 8428 tablet_bootstrap.cc:654] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:49.094761 8430 tablet_bootstrap.cc:654] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:49.095326 7902 catalog_manager.cc:4841] TS d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649): tablet 29909988929d40b0907ba97f4d5f490b (table test-table [id=13c53e458bfa43928bcb7db000db3662]) successfully deleted
I20260927 15:02:49.095439 8429 tablet_bootstrap.cc:492] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: No bootstrap required, opened a new log
I20260927 15:02:49.095525 8429 ts_tablet_manager.cc:1397] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:49.095867 8429 raft_consensus.cc:348] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.095983 8429 raft_consensus.cc:374] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:49.096024 8429 raft_consensus.cc:729] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b2f430de574935bd3cd269226b726f, State: Initialized, Role: FOLLOWER
I20260927 15:02:49.096148 8428 tablet_bootstrap.cc:492] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90: No bootstrap required, opened a new log
I20260927 15:02:49.096161 8429 consensus_queue.cc:260] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.096256 8428 ts_tablet_manager.cc:1397] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:49.096432 8429 ts_tablet_manager.cc:1428] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:49.096644 8428 raft_consensus.cc:348] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.096751 8428 raft_consensus.cc:374] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:49.096793 8428 raft_consensus.cc:729] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c29d132e0439e87df1556a51e4e90, State: Initialized, Role: FOLLOWER
I20260927 15:02:49.096911 8428 consensus_queue.cc:260] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.097167 8428 ts_tablet_manager.cc:1428] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:49.097711 8430 tablet_bootstrap.cc:492] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: No bootstrap required, opened a new log
I20260927 15:02:49.097854 8430 ts_tablet_manager.cc:1397] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 15:02:49.098187 8430 raft_consensus.cc:348] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.098265 8430 raft_consensus.cc:374] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:49.098307 8430 raft_consensus.cc:729] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17e138e781348d885de9934dde1a0fd, State: Initialized, Role: FOLLOWER
I20260927 15:02:49.098418 8430 consensus_queue.cc:260] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.098697 8430 ts_tablet_manager.cc:1428] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:02:49.165199 8448 raft_consensus.cc:482] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:02:49.165342 8448 raft_consensus.cc:504] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.165870 8448 leader_election.cc:288] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505), 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659)
I20260927 15:02:49.166343 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e063129b5ed2412ab5eadfa1916afd1f" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f" is_pre_election: true
I20260927 15:02:49.166507 8354 raft_consensus.cc:2454] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 0.
I20260927 15:02:49.166401 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e063129b5ed2412ab5eadfa1916afd1f" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17e138e781348d885de9934dde1a0fd" is_pre_election: true
I20260927 15:02:49.166646 8214 raft_consensus.cc:2454] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 0.
I20260927 15:02:49.166963 8007 leader_election.cc:302] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b2f430de574935bd3cd269226b726f, d35c29d132e0439e87df1556a51e4e90; no voters:
I20260927 15:02:49.167171 8448 raft_consensus.cc:2781] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:02:49.167263 8448 raft_consensus.cc:482] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:02:49.167341 8448 raft_consensus.cc:3037] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:49.168480 8448 raft_consensus.cc:504] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.168776 8448 leader_election.cc:288] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 election: Requested vote from peers c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505), 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659)
I20260927 15:02:49.169123 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e063129b5ed2412ab5eadfa1916afd1f" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17e138e781348d885de9934dde1a0fd"
I20260927 15:02:49.169123 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e063129b5ed2412ab5eadfa1916afd1f" candidate_uuid: "d35c29d132e0439e87df1556a51e4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b2f430de574935bd3cd269226b726f"
I20260927 15:02:49.169241 8214 raft_consensus.cc:3037] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:49.169242 8354 raft_consensus.cc:3037] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:49.170367 8214 raft_consensus.cc:2454] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 1.
I20260927 15:02:49.170380 8354 raft_consensus.cc:2454] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c29d132e0439e87df1556a51e4e90 in term 1.
I20260927 15:02:49.170657 8005 leader_election.cc:302] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c17e138e781348d885de9934dde1a0fd, d35c29d132e0439e87df1556a51e4e90; no voters:
I20260927 15:02:49.170851 8448 raft_consensus.cc:2781] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:49.171066 8448 raft_consensus.cc:686] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 1 LEADER]: Becoming Leader. State: Replica: d35c29d132e0439e87df1556a51e4e90, State: Running, Role: LEADER
I20260927 15:02:49.171224 8448 consensus_queue.cc:237] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } }
I20260927 15:02:49.172853 7916 catalog_manager.cc:5495] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c29d132e0439e87df1556a51e4e90 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d35c29d132e0439e87df1556a51e4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c29d132e0439e87df1556a51e4e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 } health_report { overall_health: UNKNOWN } } }
I20260927 15:02:49.206190 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:49.203108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.0.98.1:39649)
0927 15:02:49.203228 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:49.203235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:49.203249 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:49.203390 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:49.203393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:49.203443 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:49.203524 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:49.203757 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.204282 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.205104 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.205344 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.205458 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:49.205707 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:49.205827 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:49.205908 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:49.205999 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 15:02:49.206050 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:02:49.215410 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:49.212400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46569 (local address 127.0.98.3:41659)
0927 15:02:49.212532 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:49.212537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:49.212726 (+ 189us) server_negotiation.cc:407] Connection header received
0927 15:02:49.212838 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:49.212840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:49.212882 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:49.212944 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:49.213208 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.213685 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.214548 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.214771 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.214851 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:49.215079 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:49.215140 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:49.215192 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:49.215271 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:02:49.215310 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:02:49.215425 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:49.212316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41659 (local address 127.0.98.1:46569)
0927 15:02:49.212642 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:49.212655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:49.212744 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:49.212990 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:49.212993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:49.213001 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:49.213114 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:49.213119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.213825 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.213828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:49.214438 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:49.214446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.214867 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.214869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:49.214929 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:49.214967 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:49.215209 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:49.215214 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:02:49.215258 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 15:02:49.215319 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
I20260927 15:02:49.220561 8214 raft_consensus.cc:1261] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Refusing update from remote peer d35c29d132e0439e87df1556a51e4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:49.220604 8354 raft_consensus.cc:1261] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Refusing update from remote peer d35c29d132e0439e87df1556a51e4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:02:49.221048 8448 consensus_queue.cc:1035] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17e138e781348d885de9934dde1a0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:49.221233 8510 consensus_queue.cc:1035] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b2f430de574935bd3cd269226b726f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:02:49.236734 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:49.233675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48021 (local address 127.0.98.1:39649)
0927 15:02:49.233788 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:49.233792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:49.233848 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:02:49.233986 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:49.233988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:49.234031 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:49.234133 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:02:49.234364 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.234910 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.235662 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.235944 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.236043 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:49.236300 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:02:49.236376 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:02:49.236444 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:49.236565 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 15:02:49.236608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:02:49.236785 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:49.233609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39649 (local address 127.0.98.3:48021)
0927 15:02:49.233763 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:49.233808 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:49.233889 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:49.234145 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:49.234148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:49.234155 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:02:49.234248 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:49.234254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.235008 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.235011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:49.235556 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:49.235564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:49.236058 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:49.236062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:49.236149 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:49.236181 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:02:49.236472 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:02:49.236479 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:02:49.236563 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:02:49.236648 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 15:02:49.256430 7916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:49.256597 7916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:02:49.258641 7916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41f6ff9664134b4f9ff96687b0ed4485: Sending DeleteTablet for 3 replicas of tablet e063129b5ed2412ab5eadfa1916afd1f
I20260927 15:02:49.259173 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet e063129b5ed2412ab5eadfa1916afd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53676
I20260927 15:02:49.259255 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet e063129b5ed2412ab5eadfa1916afd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39674
I20260927 15:02:49.259346 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet e063129b5ed2412ab5eadfa1916afd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:02:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50478
I20260927 15:02:49.259364 8454 tablet_replica.cc:330] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90: stopping tablet replica
I20260927 15:02:49.259464 8452 tablet_replica.cc:330] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: stopping tablet replica
I20260927 15:02:49.259505 8453 tablet_replica.cc:330] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: stopping tablet replica
I20260927 15:02:49.259511 8454 raft_consensus.cc:2229] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:02:49.259580 8452 raft_consensus.cc:2229] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:49.259620 8453 raft_consensus.cc:2229] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:02:49.259694 8452 raft_consensus.cc:2258] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:49.259727 8454 raft_consensus.cc:2258] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:49.259732 8453 raft_consensus.cc:2258] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:02:49.260452 8454 ts_tablet_manager.cc:1905] T e063129b5ed2412ab5eadfa1916afd1f P d35c29d132e0439e87df1556a51e4e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:49.260653 8452 ts_tablet_manager.cc:1905] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:49.262449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 7989
I20260927 15:02:49.263125 8452 ts_tablet_manager.cc:1918] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:49.263209 8452 log.cc:1196] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/e063129b5ed2412ab5eadfa1916afd1f
I20260927 15:02:49.263465 8452 ts_tablet_manager.cc:1939] T e063129b5ed2412ab5eadfa1916afd1f P c17e138e781348d885de9934dde1a0fd: Deleting consensus metadata
I20260927 15:02:49.263926 8453 ts_tablet_manager.cc:1905] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:02:49.265892 7901 catalog_manager.cc:4841] TS c17e138e781348d885de9934dde1a0fd (127.0.98.2:33505): tablet e063129b5ed2412ab5eadfa1916afd1f (table test-table [id=367cb1dff25948a79577c594f1cd35c7]) successfully deleted
I20260927 15:02:49.266414 8453 ts_tablet_manager.cc:1918] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:02:49.266511 8453 log.cc:1196] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/e063129b5ed2412ab5eadfa1916afd1f
I20260927 15:02:49.266796 8453 ts_tablet_manager.cc:1939] T e063129b5ed2412ab5eadfa1916afd1f P 36b2f430de574935bd3cd269226b726f: Deleting consensus metadata
I20260927 15:02:49.267505 7903 catalog_manager.cc:4841] TS 36b2f430de574935bd3cd269226b726f (127.0.98.3:41659): tablet e063129b5ed2412ab5eadfa1916afd1f (table test-table [id=367cb1dff25948a79577c594f1cd35c7]) successfully deleted
W20260927 15:02:49.271762 7902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39649 (error 108)
I20260927 15:02:49.271821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 8129
I20260927 15:02:49.273947 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:49.273518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39649 (local address 127.0.0.1:53696)
0927 15:02:49.273757 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:49.273858 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
W20260927 15:02:49.274071 7902 catalog_manager.cc:4584] TS d35c29d132e0439e87df1556a51e4e90 (127.0.98.1:39649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e063129b5ed2412ab5eadfa1916afd1f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39649: connect: Connection refused (error 111)
I20260927 15:02:49.278781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 8269
I20260927 15:02:49.285293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 7885
2026-09-27T15:02:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3104 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:49 dist-test-slave-ml7t krb5kdc[8522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:49 dist-test-slave-ml7t krb5kdc[8522](info): set up 2 sockets
Sep 27 15:02:49 dist-test-slave-ml7t krb5kdc[8522](info): commencing operation
krb5kdc: starting...
W20260927 15:02:50.339155 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:50 dist-test-slave-ml7t krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:50.467231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:50.580471 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:50.580924 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:50.581018 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:50.585325 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:50.585404 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:50.585425 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:50.585443 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:50.585459 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:50.591033 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:50.592089 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:50.593052 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:50.601572 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:50.601578 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:50.601578 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:50.601704 8535 server_base.cc:1034] running on GCE node
I20260927 15:02:50.602396 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:50.603528 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:50.604751 8535 hybrid_clock.cc:648] HybridClock initialized: now 1790521370604718 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T15:02:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1329 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:02:50 dist-test-slave-ml7t krb5kdc[8548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:02:50 dist-test-slave-ml7t krb5kdc[8548](info): set up 2 sockets
Sep 27 15:02:50 dist-test-slave-ml7t krb5kdc[8548](info): commencing operation
krb5kdc: starting...
W20260927 15:02:51.678723 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:02:51 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:02:51.806147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:51.917531 8561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:51.917933 8561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:51.917984 8561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:51.922061 8561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:51.922130 8561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:51.922150 8561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:51.922168 8561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:51.922183 8561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:51.927847 8561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:51.928983 8561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:51.930038 8561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:51.939101 8566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:51.939530 8569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:51.939738 8567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:51.939713 8561 server_base.cc:1034] running on GCE node
I20260927 15:02:51.940372 8561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:51.941614 8561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:51.942833 8561 hybrid_clock.cc:648] HybridClock initialized: now 1790521371942819 us; error 29 us; skew 500 ppm
Sep 27 15:02:51 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:51.946275 8561 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:51.947281 8561 webserver.cc:460] Webserver started at http://127.0.98.62:34803/ using document root <none> and password file <none>
I20260927 15:02:51.947868 8561 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:51.947934 8561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:51.948128 8561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:51.950453 8561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "daafcc876e6c4aab87d5576dff82e98c"
format_stamp: "Formatted at 2026-09-27 15:02:51 on dist-test-slave-ml7t"
server_key: "3e39a7cbf5e3f816b68d04450288e919"
server_key_iv: "d54f79fe7ae00531a8599e677ee243d5"
server_key_version: "encryptionkey@0"
I20260927 15:02:51.951022 8561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "daafcc876e6c4aab87d5576dff82e98c"
format_stamp: "Formatted at 2026-09-27 15:02:51 on dist-test-slave-ml7t"
server_key: "3e39a7cbf5e3f816b68d04450288e919"
server_key_iv: "d54f79fe7ae00531a8599e677ee243d5"
server_key_version: "encryptionkey@0"
I20260927 15:02:51.955271 8561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:02:51.958204 8576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:51.959663 8561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:51.959796 8561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "daafcc876e6c4aab87d5576dff82e98c"
format_stamp: "Formatted at 2026-09-27 15:02:51 on dist-test-slave-ml7t"
server_key: "3e39a7cbf5e3f816b68d04450288e919"
server_key_iv: "d54f79fe7ae00531a8599e677ee243d5"
server_key_version: "encryptionkey@0"
I20260927 15:02:51.959899 8561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:51.981164 8561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:51.984788 8561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:51.984954 8561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:51.992470 8561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43707
I20260927 15:02:51.992493 8628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43707 every 8 connection(s)
I20260927 15:02:51.993521 8561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:51.996477 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:02:52.002137 8629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Bootstrap starting.
I20260927 15:02:52.003018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 8561
I20260927 15:02:52.003156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:02:52.003471 392 external_mini_cluster.cc:1428] Setting key 14138de1dfc9d23c9ca72e6f28a2c333
I20260927 15:02:52.005221 8629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Neither blocks nor log segments found. Creating new log.
I20260927 15:02:52.006091 8629 log.cc:824] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:52.008037 8629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: No bootstrap required, opened a new log
I20260927 15:02:52.012100 8629 raft_consensus.cc:348] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:52.012400 8629 raft_consensus.cc:374] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:02:52.012493 8629 raft_consensus.cc:729] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daafcc876e6c4aab87d5576dff82e98c, State: Initialized, Role: FOLLOWER
Sep 27 15:02:52 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:52.013013 8629 consensus_queue.cc:260] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:52.013211 8629 raft_consensus.cc:388] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:52.013298 8629 raft_consensus.cc:482] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:52.013399 8629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:02:52.014539 8629 raft_consensus.cc:504] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:52.014958 8629 leader_election.cc:302] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daafcc876e6c4aab87d5576dff82e98c; no voters:
I20260927 15:02:52.015376 8629 leader_election.cc:288] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:02:52.015508 8634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:02:52.015718 8634 raft_consensus.cc:686] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 LEADER]: Becoming Leader. State: Replica: daafcc876e6c4aab87d5576dff82e98c, State: Running, Role: LEADER
I20260927 15:02:52.016139 8634 consensus_queue.cc:237] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:52.016824 8629 sys_catalog.cc:564] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:52.018033 8636 sys_catalog.cc:455] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader daafcc876e6c4aab87d5576dff82e98c. Latest consensus state: current_term: 1 leader_uuid: "daafcc876e6c4aab87d5576dff82e98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } } }
I20260927 15:02:52.018174 8636 sys_catalog.cc:458] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:52.018474 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daafcc876e6c4aab87d5576dff82e98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } } }
I20260927 15:02:52.018568 8643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:52.018569 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:52.020370 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:52.004738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.98.62:43707)
0927 15:02:52.005217 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:52.005233 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:52.005303 (+ 70us) server_negotiation.cc:407] Connection header received
0927 15:02:52.006128 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:52.006147 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:52.006552 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:52.006918 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:52.007832 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.008999 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.010135 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.010659 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.010858 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:52.013847 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:52.013855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:52.013878 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:52.013911 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:52.016248 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.016839 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.016844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.016850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.016992 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.017329 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.017333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.017335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.017940 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:52.018132 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:52.018958 (+ 826us) server_negotiation.cc:299] Negotiation successful
0927 15:02:52.019302 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
I20260927 15:02:52.022089 8643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:52.027415 8643 catalog_manager.cc:1347] Generated new cluster ID: 29bc77f9c534406ba8443125ad3b211a
I20260927 15:02:52.027513 8643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:52.060681 8643 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:02:52.062081 8643 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:52.071259 8643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Generated new TSK 0
I20260927 15:02:52.072154 8643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:02:52.106101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43707
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:52.220100 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:52.220395 8655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:52.220489 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:52.224805 8655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:52.224886 8655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:52.224994 8655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:52.230702 8655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:52.231899 8655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:52.233021 8655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:52.242827 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:52.242827 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:52.242887 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:52.243139 8655 server_base.cc:1034] running on GCE node
I20260927 15:02:52.243650 8655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:52.244288 8655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:52.245478 8655 hybrid_clock.cc:648] HybridClock initialized: now 1790521372245447 us; error 46 us; skew 500 ppm
Sep 27 15:02:52 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:52.248588 8655 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:02:52.249564 8655 webserver.cc:460] Webserver started at http://127.0.98.1:37399/ using document root <none> and password file <none>
I20260927 15:02:52.250234 8655 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:52.250334 8655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:52.250553 8655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:52.252616 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "6fc5f14ed23248f5b3aa9245fcb8c2b2"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "6339ca395f7a6cbb1bebdad6bc3614a6"
server_key_iv: "78dcb0c8d13cec04d49a60ffc53ca954"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.253172 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6fc5f14ed23248f5b3aa9245fcb8c2b2"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "6339ca395f7a6cbb1bebdad6bc3614a6"
server_key_iv: "78dcb0c8d13cec04d49a60ffc53ca954"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.256994 8655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 15:02:52.259550 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.260859 8655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:02:52.261036 8655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "6fc5f14ed23248f5b3aa9245fcb8c2b2"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "6339ca395f7a6cbb1bebdad6bc3614a6"
server_key_iv: "78dcb0c8d13cec04d49a60ffc53ca954"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.261161 8655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:52.271942 8655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:52.275110 8655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:52.275341 8655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:52.275977 8655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:52.276862 8655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:52.276953 8655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.277021 8655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:52.277077 8655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.286871 8655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38827
I20260927 15:02:52.286901 8783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38827 every 8 connection(s)
I20260927 15:02:52.287832 8655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:52.292953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 8655
I20260927 15:02:52.293109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:02:52.293408 392 external_mini_cluster.cc:1428] Setting key 4913e0137550469131c1f0fc961c3e8c
Sep 27 15:02:52 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:02:52.303836 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:52.289862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36603 (local address 127.0.98.62:43707)
0927 15:02:52.290023 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:52.290028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:52.290953 (+ 925us) server_negotiation.cc:407] Connection header received
0927 15:02:52.292272 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:52.292275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:52.292323 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:52.292407 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:52.294059 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.294830 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.296583 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.296998 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.297120 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:52.299903 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:52.299907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:52.299910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:52.299939 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:52.301502 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.302204 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.302208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.302210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.302292 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.302664 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.302667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.302669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.302890 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:52.302999 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:52.303528 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 15:02:52.303659 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:02:52.304687 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:52.290332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:36603)
0927 15:02:52.290787 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:52.290829 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:52.292059 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:52.292639 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:52.292649 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:52.293174 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:52.293897 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:52.293915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.294962 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.294968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:52.296412 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:52.296426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.297152 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.297155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:52.297291 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:52.297934 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:52.297951 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:52.299723 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:52.301695 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:52.301701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:52.301717 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:52.302072 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:52.302400 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:52.302405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:52.302407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:52.302553 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:52.303020 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:52.303026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:52.303277 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 15:02:52.303474 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 15:02:52.303803 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:02:52.306162 8784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43707
I20260927 15:02:52.306427 8784 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:52.306922 8784 heartbeater.cc:511] Master 127.0.98.62:43707 requested a full tablet report, sending...
I20260927 15:02:52.308488 8593 ts_manager.cc:194] Registered new tserver with Master: 6fc5f14ed23248f5b3aa9245fcb8c2b2 (127.0.98.1:38827)
I20260927 15:02:52.310063 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:02:52.322055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43707
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:52.463446 8789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:52.464026 8789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:52.464078 8789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:52.469075 8789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:52.469154 8789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:52.469224 8789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:52.475230 8789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:52.476356 8789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:52.477439 8789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:52.487828 8795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:52.487828 8794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:52.487924 8797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:52.488258 8789 server_base.cc:1034] running on GCE node
I20260927 15:02:52.488734 8789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:52.489428 8789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:52.490638 8789 hybrid_clock.cc:648] HybridClock initialized: now 1790521372490598 us; error 66 us; skew 500 ppm
Sep 27 15:02:52 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:52.493518 8789 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:02:52.494503 8789 webserver.cc:460] Webserver started at http://127.0.98.2:34351/ using document root <none> and password file <none>
I20260927 15:02:52.495097 8789 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:52.495194 8789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:52.495448 8789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:52.497553 8789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "e9bd311ec6ef4281be332152e58a67b1"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "d6b37fc00b245458333084b586e32aab"
server_key_iv: "d4f80ebad970d3a02bc0dfe3d9360d21"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.498198 8789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e9bd311ec6ef4281be332152e58a67b1"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "d6b37fc00b245458333084b586e32aab"
server_key_iv: "d4f80ebad970d3a02bc0dfe3d9360d21"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.502321 8789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:02:52.504935 8804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.506284 8789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:02:52.506425 8789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "e9bd311ec6ef4281be332152e58a67b1"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "d6b37fc00b245458333084b586e32aab"
server_key_iv: "d4f80ebad970d3a02bc0dfe3d9360d21"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.506558 8789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:52.517946 8789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:52.521313 8789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:52.521535 8789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:52.522207 8789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:52.523058 8789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:52.523140 8789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.523231 8789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:52.523281 8789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.532444 8789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46165
I20260927 15:02:52.532466 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46165 every 8 connection(s)
I20260927 15:02:52.533387 8789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:52.539305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 8789
I20260927 15:02:52.539443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:02:52.539733 392 external_mini_cluster.cc:1428] Setting key fc9955ea210e7e72191aae9facc90081
Sep 27 15:02:52 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:52.549839 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:52.535348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54835 (local address 127.0.98.62:43707)
0927 15:02:52.535505 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:52.535510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:52.536495 (+ 985us) server_negotiation.cc:407] Connection header received
0927 15:02:52.537941 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:52.537944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:52.537991 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:52.538078 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:52.539358 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.540192 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.541759 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.542181 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.542305 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:52.545049 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:52.545055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:52.545062 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:52.545095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:52.547310 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.547959 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.547963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.547966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.548072 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.548429 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.548432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.548434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.548708 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:52.548823 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:52.549402 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 15:02:52.549557 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:02:52.550709 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:52.535856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:54835)
0927 15:02:52.536284 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:52.536325 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:52.537689 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:52.538252 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:52.538259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:52.538645 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:52.539187 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:52.539201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.540342 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.540345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:52.541552 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:52.541616 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.542339 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.542342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:52.542480 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:52.543194 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:52.543210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:52.544852 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:52.547498 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:52.547503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:52.547545 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:52.547835 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:52.548178 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:52.548180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:52.548182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:52.548323 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:52.548837 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:52.548843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:52.549137 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 15:02:52.549307 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 15:02:52.549639 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
I20260927 15:02:52.552016 8918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43707
I20260927 15:02:52.552291 8918 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:52.552907 8918 heartbeater.cc:511] Master 127.0.98.62:43707 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:02:52.554265 8593 ts_manager.cc:194] Registered new tserver with Master: e9bd311ec6ef4281be332152e58a67b1 (127.0.98.2:46165)
I20260927 15:02:52.555004 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:02:52.572762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43707
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:52.687043 8923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:52.687376 8923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:52.687472 8923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:52.691639 8923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:52.691761 8923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:52.691859 8923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:52.697299 8923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:52.698470 8923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:52.699571 8923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:52.709414 8929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:52.709404 8931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:52.709404 8928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:52.709790 8923 server_base.cc:1034] running on GCE node
I20260927 15:02:52.710198 8923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:52.710968 8923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:52.712162 8923 hybrid_clock.cc:648] HybridClock initialized: now 1790521372712140 us; error 39 us; skew 500 ppm
Sep 27 15:02:52 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:52.715130 8923 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:02:52.715974 8923 webserver.cc:460] Webserver started at http://127.0.98.3:46073/ using document root <none> and password file <none>
I20260927 15:02:52.716636 8923 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:52.716702 8923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:52.716917 8923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:02:52.719199 8923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2db326b7c5845b8b3ef95a82ac2564e"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "08587a1ffff0133df82eb199a2354846"
server_key_iv: "1620f9528125f9bea1fbf803097589d3"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.719751 8923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2db326b7c5845b8b3ef95a82ac2564e"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "08587a1ffff0133df82eb199a2354846"
server_key_iv: "1620f9528125f9bea1fbf803097589d3"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.723588 8923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:02:52.726194 8938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.727480 8923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:52.727571 8923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "f2db326b7c5845b8b3ef95a82ac2564e"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "08587a1ffff0133df82eb199a2354846"
server_key_iv: "1620f9528125f9bea1fbf803097589d3"
server_key_version: "encryptionkey@0"
I20260927 15:02:52.727711 8923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:52.750757 8923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:52.754429 8923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:52.754659 8923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:52.755324 8923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:52.756232 8923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:52.756281 8923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.756353 8923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:52.756392 8923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:52.765689 8923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38373
I20260927 15:02:52.765771 9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38373 every 8 connection(s)
I20260927 15:02:52.766767 8923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:52.769615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 8923
I20260927 15:02:52.769755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:02:52.770087 392 external_mini_cluster.cc:1428] Setting key 22725035d5da3917d2049bb3881f626c
Sep 27 15:02:52 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:02:52.781647 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:52.768711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55073 (local address 127.0.98.62:43707)
0927 15:02:52.768883 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:52.768888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:52.769953 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 15:02:52.771713 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:52.771717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:52.771773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:52.771891 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:52.773269 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.773921 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.775148 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.775380 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.775501 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:52.778170 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:52.778174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:52.778176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:52.778201 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:52.779565 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.780086 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.780089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.780090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.780175 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:52.780533 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:52.780536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:52.780537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:52.780728 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:52.780809 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:52.781358 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 15:02:52.781521 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:02:52.782554 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:52.769177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:55073)
0927 15:02:52.769745 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:52.769830 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:52.771497 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:52.772086 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:52.772093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:52.772586 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:52.773111 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:52.773125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.774057 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.774065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:52.775003 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:52.775013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:52.775545 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:52.775549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:52.775755 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:52.776332 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:52.776350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:52.778000 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:02:52.779675 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:52.779681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:52.779696 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:52.779969 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:52.780267 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:02:52.780269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:02:52.780271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:02:52.780414 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:02:52.780830 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:02:52.780835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:02:52.781103 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 15:02:52.781286 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 15:02:52.781609 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":160,"threads_started":1}
I20260927 15:02:52.783918 9052 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43707
I20260927 15:02:52.784241 9052 heartbeater.cc:464] Registering TS with master...
I20260927 15:02:52.785014 9052 heartbeater.cc:511] Master 127.0.98.62:43707 requested a full tablet report, sending...
I20260927 15:02:52.786330 8589 ts_manager.cc:194] Registered new tserver with Master: f2db326b7c5845b8b3ef95a82ac2564e (127.0.98.3:38373)
I20260927 15:02:52.786896 8589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55073
I20260927 15:02:52.796535 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 15:02:52.874723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 8655
I20260927 15:02:52.881013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 8789
I20260927 15:02:52.886582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 8923
I20260927 15:02:52.892048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 8561
I20260927 15:02:52.898227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43707
--webserver_interface=127.0.98.62
--webserver_port=34803
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:02:53.008294 9061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:53.008602 9061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:53.008646 9061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:53.012916 9061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:02:53.012984 9061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:53.013003 9061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:53.013020 9061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:02:53.013036 9061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:02:53.018481 9061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:53.019546 9061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:53.020573 9061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:53.028957 9067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:53.029031 9061 server_base.cc:1034] running on GCE node
W20260927 15:02:53.028990 9069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:53.028983 9066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:53.029897 9061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:53.031086 9061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:53.032291 9061 hybrid_clock.cc:648] HybridClock initialized: now 1790521373032272 us; error 31 us; skew 500 ppm
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:53.035177 9061 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:02:53.036031 9061 webserver.cc:460] Webserver started at http://127.0.98.62:34803/ using document root <none> and password file <none>
I20260927 15:02:53.036818 9061 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:53.036939 9061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:53.040802 9061 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:02:53.043047 9076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.044405 9061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:53.044625 9061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "daafcc876e6c4aab87d5576dff82e98c"
format_stamp: "Formatted at 2026-09-27 15:02:51 on dist-test-slave-ml7t"
server_key: "3e39a7cbf5e3f816b68d04450288e919"
server_key_iv: "d54f79fe7ae00531a8599e677ee243d5"
server_key_version: "encryptionkey@0"
I20260927 15:02:53.045226 9061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:53.058898 9061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:53.062388 9061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:53.062618 9061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:53.070225 9061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43707
I20260927 15:02:53.070276 9128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43707 every 8 connection(s)
I20260927 15:02:53.071336 9061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:02:53.074627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9061
I20260927 15:02:53.075146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38827
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43707
--webserver_port=37399
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:02:53.076736 9129 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:02:53.078526 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Bootstrap starting.
I20260927 15:02:53.095126 9129 log.cc:824] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:02:53.101964 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:02:53.102355 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Bootstrap complete.
I20260927 15:02:53.106559 9129 raft_consensus.cc:348] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:53.106941 9129 raft_consensus.cc:729] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: daafcc876e6c4aab87d5576dff82e98c, State: Initialized, Role: FOLLOWER
I20260927 15:02:53.107434 9129 consensus_queue.cc:260] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:53.107544 9129 raft_consensus.cc:388] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:02:53.107640 9129 raft_consensus.cc:482] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:02:53.107739 9129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:02:53.109709 9129 raft_consensus.cc:504] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:53.110059 9129 leader_election.cc:302] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daafcc876e6c4aab87d5576dff82e98c; no voters:
I20260927 15:02:53.110425 9129 leader_election.cc:288] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:02:53.110502 9133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:02:53.110679 9133 raft_consensus.cc:686] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [term 2 LEADER]: Becoming Leader. State: Replica: daafcc876e6c4aab87d5576dff82e98c, State: Running, Role: LEADER
I20260927 15:02:53.110965 9133 consensus_queue.cc:237] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } }
I20260927 15:02:53.111624 9129 sys_catalog.cc:564] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:02:53.112360 9134 sys_catalog.cc:455] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "daafcc876e6c4aab87d5576dff82e98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } } }
I20260927 15:02:53.112491 9134 sys_catalog.cc:458] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:53.113060 9135 sys_catalog.cc:455] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader daafcc876e6c4aab87d5576dff82e98c. Latest consensus state: current_term: 2 leader_uuid: "daafcc876e6c4aab87d5576dff82e98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daafcc876e6c4aab87d5576dff82e98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43707 } } }
I20260927 15:02:53.113204 9135 sys_catalog.cc:458] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c [sys.catalog]: This master's current role is: LEADER
I20260927 15:02:53.113487 9142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:02:53.116837 9142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:02:53.117456 9142 catalog_manager.cc:1259] Loaded cluster ID: 29bc77f9c534406ba8443125ad3b211a
I20260927 15:02:53.117515 9142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:02:53.119943 9142 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:02:53.120714 9142 catalog_manager.cc:5879] T 00000000000000000000000000000000 P daafcc876e6c4aab87d5576dff82e98c: Loaded TSK: 0
I20260927 15:02:53.122202 9142 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:02:53.192584 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:53.192839 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:53.192883 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:53.196982 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:53.197052 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:53.197111 9131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:02:53.202540 9131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37399
--tserver_master_addrs=127.0.98.62:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:53.203567 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:53.204586 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:53.215516 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:53.215458 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:53.215534 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:53.216203 9131 server_base.cc:1034] running on GCE node
I20260927 15:02:53.216694 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:53.217432 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:53.218644 9131 hybrid_clock.cc:648] HybridClock initialized: now 1790521373218633 us; error 28 us; skew 500 ppm
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:53.221601 9131 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 15:02:53.222631 9131 webserver.cc:460] Webserver started at http://127.0.98.1:37399/ using document root <none> and password file <none>
I20260927 15:02:53.223206 9131 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:53.223277 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:53.227555 9131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 15:02:53.229704 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.231038 9131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:02:53.231135 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "6fc5f14ed23248f5b3aa9245fcb8c2b2"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "6339ca395f7a6cbb1bebdad6bc3614a6"
server_key_iv: "78dcb0c8d13cec04d49a60ffc53ca954"
server_key_version: "encryptionkey@0"
I20260927 15:02:53.231714 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:53.252710 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:53.255998 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:53.256210 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:53.256855 9131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:53.257701 9131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:53.257747 9131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.257864 9131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:53.257898 9131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.267372 9131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38827
I20260927 15:02:53.267408 9278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38827 every 8 connection(s)
I20260927 15:02:53.268405 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:02:53.272377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9131
I20260927 15:02:53.272872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46165
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43707
--webserver_port=34351
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:53.283608 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.270760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:38663)
0927 15:02:53.271252 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.271293 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.272891 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.274182 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.274192 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.274559 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.275018 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.275031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.276398 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.276403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.278056 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.278070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.278829 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.278836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.279064 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.279711 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.279738 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.282625 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
W20260927 15:02:53.283595 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.270617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38663 (local address 127.0.98.62:43707)
0927 15:02:53.271122 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.271143 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.271408 (+ 265us) server_negotiation.cc:407] Connection header received
0927 15:02:53.273375 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.273388 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.273691 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.274096 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.275195 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.276221 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.278200 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.278656 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.278774 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.282850 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38663: BlockingRecv error: recv got EOF from 127.0.98.1:38663 (error 108)
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
W20260927 15:02:53.284273 9279 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43707 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43707: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:53.290114 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.284759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:57829)
0927 15:02:53.284859 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.284876 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.284962 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.285235 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.285238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.285419 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.285512 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.285517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.286363 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.286366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.286939 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.286948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.287443 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.287447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.287550 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.288115 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.288127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.289956 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 15:02:53.290208 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.284840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57829 (local address 127.0.98.62:43707)
0927 15:02:53.284976 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.284982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.285002 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:02:53.285056 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.285059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.285108 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.285223 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.285636 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.286236 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.287069 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.287317 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.287442 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.290082 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57829: BlockingRecv error: recv got EOF from 127.0.98.1:57829 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:53.296414 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.290813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:50187)
0927 15:02:53.291019 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.291033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.291102 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.291321 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.291323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.291451 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.291521 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.291526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.292556 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.292559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.293074 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.293081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.293651 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.293653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.293709 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.294333 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.294344 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.296291 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260927 15:02:53.296612 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.290963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50187 (local address 127.0.98.62:43707)
0927 15:02:53.291084 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.291088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.291102 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:53.291181 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.291184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.291226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.291298 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.291628 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.292416 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.293193 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.293532 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.293646 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.296441 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50187: BlockingRecv error: recv got EOF from 127.0.98.1:50187 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:02:53.296640 9279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:02:53.388958 9283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:53.389281 9283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:53.389369 9283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:53.393486 9283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:53.393602 9283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:53.393708 9283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:02:53.400761 9283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34351
--tserver_master_addrs=127.0.98.62:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:53.402012 9283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:53.403079 9283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:02:53.413295 9283 server_base.cc:1034] running on GCE node
W20260927 15:02:53.413291 9289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:53.413304 9288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:53.413293 9291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:53.414122 9283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:53.414834 9283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:53.416031 9283 hybrid_clock.cc:648] HybridClock initialized: now 1790521373415995 us; error 52 us; skew 500 ppm
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:53.419094 9283 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 15:02:53.420047 9283 webserver.cc:460] Webserver started at http://127.0.98.2:34351/ using document root <none> and password file <none>
I20260927 15:02:53.420635 9283 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:53.420737 9283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:53.424969 9283 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:02:53.427307 9298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.428561 9283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:02:53.428704 9283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "e9bd311ec6ef4281be332152e58a67b1"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "d6b37fc00b245458333084b586e32aab"
server_key_iv: "d4f80ebad970d3a02bc0dfe3d9360d21"
server_key_version: "encryptionkey@0"
I20260927 15:02:53.429251 9283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:53.440325 9283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:53.443271 9283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:53.443488 9283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:53.444065 9283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:53.444877 9283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:53.444957 9283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.445026 9283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:53.445079 9283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.454493 9283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46165
I20260927 15:02:53.454531 9411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46165 every 8 connection(s)
I20260927 15:02:53.455482 9283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:02:53.460012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9283
I20260927 15:02:53.460508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38373
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43707
--webserver_port=46073
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:02:53.467774 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.457291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59915 (local address 127.0.98.62:43707)
0927 15:02:53.457459 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.457464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.458343 (+ 879us) server_negotiation.cc:407] Connection header received
0927 15:02:53.459628 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.459631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.459678 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.459776 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.461433 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.462244 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.463357 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.463803 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.463955 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.467581 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59915: BlockingRecv error: recv got EOF from 127.0.98.2:59915 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:02:53.468218 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.457698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:59915)
0927 15:02:53.458196 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.458236 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.459401 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.459997 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.460007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.460495 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.461217 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.461238 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.462386 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.462388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.463229 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.463237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.463965 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.463968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.464099 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.464794 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.464809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.467356 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
W20260927 15:02:53.468825 9412 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43707 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43707: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:53.474754 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.469280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:41587)
0927 15:02:53.469392 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.469408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.469506 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.469826 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.469829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.470000 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.470090 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.470095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.470803 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.470809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.471355 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.471362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.471809 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.471812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.471908 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.472561 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.472584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.474656 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:02:53.475165 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.469436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41587 (local address 127.0.98.62:43707)
0927 15:02:53.469612 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.469615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.469625 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:02:53.469661 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.469663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.469701 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.469816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.470204 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.470694 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.471476 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.471697 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.471810 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.474824 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41587: BlockingRecv error: recv got EOF from 127.0.98.2:41587 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:53.480580 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.475302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:60343)
0927 15:02:53.475477 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.475490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.475603 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.475857 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.475859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.475980 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.476051 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.476057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.477064 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.477066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.477571 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.477578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.478155 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.478157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.478213 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.478722 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.478733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.480459 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 15:02:53.480801 9412 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:02:53.480952 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.475356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60343 (local address 127.0.98.62:43707)
0927 15:02:53.475477 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.475482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.475550 (+ 68us) server_negotiation.cc:407] Connection header received
0927 15:02:53.475704 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.475707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.475751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.475838 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.476162 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.476958 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.477680 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.478045 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.478156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.480641 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60343: BlockingRecv error: recv got EOF from 127.0.98.2:60343 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:02:53.581650 9415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:02:53.581998 9415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:02:53.582049 9415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:02:53.586112 9415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:02:53.586170 9415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:02:53.586227 9415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:02:53.591567 9415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46073
--tserver_master_addrs=127.0.98.62:43707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:02:53.592624 9415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:02:53.593657 9415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:02:53.603241 9421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:53.603340 9415 server_base.cc:1034] running on GCE node
W20260927 15:02:53.603287 9420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:02:53.603299 9423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:02:53.604027 9415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:02:53.604662 9415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:02:53.605846 9415 hybrid_clock.cc:648] HybridClock initialized: now 1790521373605823 us; error 30 us; skew 500 ppm
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521373, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:02:53.608718 9415 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 15:02:53.609681 9415 webserver.cc:460] Webserver started at http://127.0.98.3:46073/ using document root <none> and password file <none>
I20260927 15:02:53.610285 9415 fs_manager.cc:359] Metadata directory not provided
I20260927 15:02:53.610404 9415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:02:53.614647 9415 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:02:53.616760 9430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.618101 9415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:02:53.618237 9415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "f2db326b7c5845b8b3ef95a82ac2564e"
format_stamp: "Formatted at 2026-09-27 15:02:52 on dist-test-slave-ml7t"
server_key: "08587a1ffff0133df82eb199a2354846"
server_key_iv: "1620f9528125f9bea1fbf803097589d3"
server_key_version: "encryptionkey@0"
I20260927 15:02:53.618777 9415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:02:53.632709 9415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:02:53.636006 9415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:02:53.636216 9415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:02:53.636880 9415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:02:53.637701 9415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:02:53.637773 9415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.637863 9415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:02:53.637897 9415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:02:53.647082 9415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38373
I20260927 15:02:53.647092 9543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38373 every 8 connection(s)
I20260927 15:02:53.648094 9415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:02:53.657707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9415
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:02:53.659744 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.650260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38191 (local address 127.0.98.62:43707)
0927 15:02:53.650400 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.650404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.651204 (+ 800us) server_negotiation.cc:407] Connection header received
0927 15:02:53.652633 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.652636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.652689 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.652807 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.653964 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.654492 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.655650 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.655875 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.656006 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.659629 (+ 3623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38191: BlockingRecv error: recv got EOF from 127.0.98.3:38191 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:02:53.660573 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.650630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:38191)
0927 15:02:53.651051 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.651093 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.652375 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.652973 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.652979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.653324 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.653817 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.653830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.654621 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.654625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.655522 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.655531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.656039 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.656043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.656212 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.656784 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.656800 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.659529 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
W20260927 15:02:53.661067 9544 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43707 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43707: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:53.666327 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.661506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:36151)
0927 15:02:53.661596 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.661612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.661683 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.661928 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.661930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.662060 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.662145 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.662150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.662893 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.662899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.663424 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.663431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.663873 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.663877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.663964 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.664416 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.664426 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.666183 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 15:02:53.666405 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.661614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36151 (local address 127.0.98.62:43707)
0927 15:02:53.661713 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.661719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.661733 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:53.661784 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.661787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.661824 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.661893 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.662270 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.662779 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.663562 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.663762 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.663872 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.666290 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36151: BlockingRecv error: recv got EOF from 127.0.98.3:36151 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 15:02:53.669066 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.658812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.0.98.62:43707)
0927 15:02:53.659065 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.659068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.660015 (+ 947us) server_negotiation.cc:407] Connection header received
0927 15:02:53.660140 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.660142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.660182 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.660260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.661142 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.662119 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.662976 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.663205 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.663321 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.664878 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:02:53.664883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:02:53.664898 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:02:53.664930 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:02:53.666873 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:53.667415 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:53.667424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:53.667433 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:53.667555 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:02:53.667846 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:02:53.667851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:02:53.667853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:02:53.668205 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:02:53.668415 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:02:53.668713 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 15:02:53.668953 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":116,"threads_started":1}
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:53 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:53.672039 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:53.666919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:39689)
0927 15:02:53.667023 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:53.667038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:53.667111 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:53.667331 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:53.667333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:53.667483 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:53.667554 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.667559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.668253 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.668259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.668773 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:53.668780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.669361 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.669365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:53.669441 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.669933 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:53.669946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:53.671908 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:02:53.672325 9544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:02:53.672415 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:53.666997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39689 (local address 127.0.98.62:43707)
0927 15:02:53.667141 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:53.667146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:53.667160 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:53.667193 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:53.667196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:53.667236 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:53.667319 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:53.667664 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.668147 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.668928 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:53.669149 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:53.669259 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:53.672008 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39689: BlockingRecv error: recv got EOF from 127.0.98.3:39689 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 15:02:54 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:54 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:54.303005 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:54.297266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:43653)
0927 15:02:54.297542 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:54.297558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:54.297665 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:54.297957 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:54.297959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:54.298146 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:54.298241 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:54.298246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.299012 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.299014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:54.299599 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:54.299606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.300074 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.300077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:54.300136 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:54.300688 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:54.300698 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:54.302822 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":119,"threads_started":1}
W20260927 15:02:54.303094 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:54.297358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43653 (local address 127.0.98.62:43707)
0927 15:02:54.297687 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:54.297691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:54.297726 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:02:54.297764 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:54.297786 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:54.297831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:54.297925 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:54.298373 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.298915 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.299722 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.299964 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.300050 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:54.302960 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43653: BlockingRecv error: recv got EOF from 127.0.98.1:43653 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
Sep 27 15:02:54 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:54 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:54.487529 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:54.481567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:50143)
0927 15:02:54.481866 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:54.481881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:54.482016 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:54.482422 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:54.482425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:54.482676 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:54.482839 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:54.482846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.483916 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.483919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:54.484468 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:54.484475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.484947 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.484949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:54.485019 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:54.485547 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:54.485559 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:54.487406 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
W20260927 15:02:54.487741 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:54.481669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50143 (local address 127.0.98.62:43707)
0927 15:02:54.481868 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:54.481873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:54.481944 (+ 71us) server_negotiation.cc:407] Connection header received
0927 15:02:54.482215 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:54.482218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:54.482283 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:54.482387 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:54.483084 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.483804 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.484606 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.484843 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.484933 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:54.487583 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50143: BlockingRecv error: recv got EOF from 127.0.98.2:50143 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:02:54 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:54 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:54.678647 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:54.672949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:52461)
0927 15:02:54.673222 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:54.673234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:54.673366 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:54.673644 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:54.673647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:54.673922 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:54.674038 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:54.674046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.674801 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.674808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:54.675386 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:54.675394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.675867 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.675871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:54.675967 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:54.676554 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:54.676568 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:54.678521 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
W20260927 15:02:54.678819 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:54.672997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52461 (local address 127.0.98.62:43707)
0927 15:02:54.673171 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:54.673176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:54.673312 (+ 136us) server_negotiation.cc:407] Connection header received
0927 15:02:54.673469 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:54.673472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:54.673523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:54.673625 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:54.674165 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.674691 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.675509 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:54.675754 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:54.675869 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:54.678660 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52461: BlockingRecv error: recv got EOF from 127.0.98.3:52461 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 15:02:55 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:55 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:55.309461 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:55.303915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:59853)
0927 15:02:55.304169 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:55.304185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:55.304290 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:55.304620 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:55.304623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:55.304804 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:55.304901 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:55.304906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.305710 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.305712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:55.306347 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:55.306355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.306866 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.306868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:55.306930 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:55.307484 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:55.307495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:55.309324 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
W20260927 15:02:55.309651 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:55.303936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59853 (local address 127.0.98.62:43707)
0927 15:02:55.304242 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:55.304245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:55.304257 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:02:55.304388 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:55.304394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:55.304461 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:55.304555 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:55.305018 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.305608 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.306494 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.306751 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.306849 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:55.309493 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59853: BlockingRecv error: recv got EOF from 127.0.98.1:59853 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":137,"threads_started":1}
Sep 27 15:02:55 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:55 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:55.494225 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:55.488376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:46269)
0927 15:02:55.488673 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:55.488687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:55.488788 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:55.489042 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:55.489045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:55.489224 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:55.489326 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:55.489331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.490117 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.490120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:55.490710 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:55.490720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.491189 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.491192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:55.491258 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:55.491979 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:55.491997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:55.494019 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":116,"threads_started":1}
W20260927 15:02:55.494256 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:55.488526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46269 (local address 127.0.98.62:43707)
0927 15:02:55.488679 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:55.488682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:55.488763 (+ 81us) server_negotiation.cc:407] Connection header received
0927 15:02:55.488875 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:55.488877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:55.488920 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:55.488993 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:55.489444 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.490002 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.490826 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.491047 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.491128 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:55.494155 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46269: BlockingRecv error: recv got EOF from 127.0.98.2:46269 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:02:55 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:55 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:55.685214 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:55.679436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:59385)
0927 15:02:55.679780 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:55.679797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:55.679901 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:55.680218 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:55.680221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:55.680408 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:55.680500 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:55.680506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.681267 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.681269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:55.681851 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:55.681858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.682353 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.682357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:55.682425 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:55.683004 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:55.683018 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:55.685070 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
W20260927 15:02:55.685362 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:55.679647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59385 (local address 127.0.98.62:43707)
0927 15:02:55.679815 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:55.679824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:55.679866 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:02:55.680021 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:55.680024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:55.680080 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:55.680168 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:55.680635 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.681170 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.681993 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:55.682223 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:55.682313 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:55.685237 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59385: BlockingRecv error: recv got EOF from 127.0.98.3:59385 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 15:02:56 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:56 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:56.320026 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:56.310370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:59047)
0927 15:02:56.310668 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:56.310693 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:56.310787 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:56.311033 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:56.311035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:56.311203 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:56.311294 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:56.311299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.312553 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.312556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:56.313304 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:56.313314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.313839 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.313844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:56.313930 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:56.317642 (+ 3712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:56.317655 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:56.319831 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
W20260927 15:02:56.320130 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:56.310529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59047 (local address 127.0.98.62:43707)
0927 15:02:56.310774 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:56.310777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:56.310791 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:02:56.310869 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:56.310871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:56.310912 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:56.310990 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:56.311430 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.312303 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.313433 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.313670 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.313825 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:56.319972 (+ 6147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59047: BlockingRecv error: recv got EOF from 127.0.98.1:59047 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
Sep 27 15:02:56 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:56 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:56.500505 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:56.495191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:42543)
0927 15:02:56.495480 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:56.495494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:56.495629 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:56.495887 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:56.495889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:56.496070 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:56.496162 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:56.496167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.496906 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.496908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:56.497460 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:56.497467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.497897 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.497899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:56.497959 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:56.498442 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:56.498453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:56.500361 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
W20260927 15:02:56.500669 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:56.495236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42543 (local address 127.0.98.62:43707)
0927 15:02:56.495391 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:56.495396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:56.495591 (+ 195us) server_negotiation.cc:407] Connection header received
0927 15:02:56.495747 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:56.495750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:56.495802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:56.495870 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:56.496259 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.496813 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.497561 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.497799 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.497871 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:56.500511 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42543: BlockingRecv error: recv got EOF from 127.0.98.2:42543 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 15:02:56 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:56 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:56.691828 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:56.686051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:46507)
0927 15:02:56.686383 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:56.686402 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:56.686546 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:56.686832 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:56.686835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:56.687051 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:56.687150 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:56.687156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.687972 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.687975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:56.688532 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:56.688539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.689006 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.689010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:56.689093 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:56.689647 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:56.689658 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:56.691694 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":84,"threads_started":1}
W20260927 15:02:56.692099 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:56.686178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46507 (local address 127.0.98.62:43707)
0927 15:02:56.686340 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:56.686344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:56.686501 (+ 157us) server_negotiation.cc:407] Connection header received
0927 15:02:56.686655 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:56.686659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:56.686716 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:56.686810 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:56.687274 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.687822 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.688659 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:56.688886 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:56.689009 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:56.691845 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46507: BlockingRecv error: recv got EOF from 127.0.98.3:46507 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 15:02:57 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:57 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:57.326534 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:57.320913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:49915)
0927 15:02:57.321166 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:57.321180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:57.321267 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:57.321569 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:57.321572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:57.321813 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:57.321920 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:57.321927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.322815 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.322817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:57.323432 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:57.323442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.323867 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.323869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:57.323930 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:57.324446 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:57.324457 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:57.326407 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
W20260927 15:02:57.326731 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:57.320959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49915 (local address 127.0.98.62:43707)
0927 15:02:57.321206 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:57.321209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:57.321225 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:02:57.321382 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:57.321386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:57.321437 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:57.321535 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:57.322105 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.322721 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.323549 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.323771 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.323855 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:57.326558 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49915: BlockingRecv error: recv got EOF from 127.0.98.1:49915 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
Sep 27 15:02:57 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:57 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:57.507018 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:57.501400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:53151)
0927 15:02:57.501703 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:57.501721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:57.501873 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:57.502166 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:57.502168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:57.502338 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:57.502429 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:57.502434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.503141 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.503144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:57.503866 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:57.503876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.504460 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.504463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:57.504559 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:57.505035 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:57.505046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:57.506838 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":80,"threads_started":1}
W20260927 15:02:57.507074 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:57.501471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53151 (local address 127.0.98.62:43707)
0927 15:02:57.501686 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:57.501690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:57.501793 (+ 103us) server_negotiation.cc:407] Connection header received
0927 15:02:57.501968 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:57.501970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:57.502053 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:57.502142 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:57.502532 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.503052 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.503981 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.504346 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.504461 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:57.506956 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53151: BlockingRecv error: recv got EOF from 127.0.98.2:53151 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 27 15:02:57 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:57 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:57.698251 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:57.692674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:53133)
0927 15:02:57.693019 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:57.693032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:57.693135 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:57.693355 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:57.693357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:57.693547 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:57.693639 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:57.693644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.694555 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.694557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:57.695115 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:57.695124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.695587 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.695590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:57.695666 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:57.696189 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:57.696202 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:57.698123 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
W20260927 15:02:57.698431 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:57.692799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53133 (local address 127.0.98.62:43707)
0927 15:02:57.692933 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:57.692936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:57.693101 (+ 165us) server_negotiation.cc:407] Connection header received
0927 15:02:57.693203 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:57.693205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:57.693250 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:57.693319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:57.693736 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.694399 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.695220 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:57.695430 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:57.695537 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:57.698266 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53133: BlockingRecv error: recv got EOF from 127.0.98.3:53133 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 15:02:58 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:58 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:58.333070 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:58.327299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:45361)
0927 15:02:58.327623 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:58.327637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:58.327735 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:58.327984 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:58.327986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:58.328178 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:58.328270 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:58.328275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.329016 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.329019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:58.329576 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:58.329583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.330151 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.330154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:58.330250 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:58.330847 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:58.330861 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:58.332902 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
W20260927 15:02:58.333117 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:58.327389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45361 (local address 127.0.98.62:43707)
0927 15:02:58.327636 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:58.327639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:58.327708 (+ 69us) server_negotiation.cc:407] Connection header received
0927 15:02:58.327813 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:58.327815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:58.327857 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:58.327956 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:58.328363 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.328916 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.329693 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.330045 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.330147 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:58.333015 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45361: BlockingRecv error: recv got EOF from 127.0.98.1:45361 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":59,"threads_started":1}
Sep 27 15:02:58 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:58 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:58.513284 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:58.507805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:56781)
0927 15:02:58.508081 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:58.508095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:58.508184 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:58.508412 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:58.508415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:58.508590 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:58.508680 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:58.508685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.509470 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.509472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:58.510096 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:58.510105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.510579 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.510581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:58.510643 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:58.511154 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:58.511165 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:58.513159 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
W20260927 15:02:58.513552 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:58.507948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56781 (local address 127.0.98.62:43707)
0927 15:02:58.508099 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:58.508102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:58.508168 (+ 66us) server_negotiation.cc:407] Connection header received
0927 15:02:58.508261 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:58.508263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:58.508308 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:58.508383 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:58.508802 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.509362 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.510230 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.510467 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.510557 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:58.513280 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56781: BlockingRecv error: recv got EOF from 127.0.98.2:56781 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 15:02:58 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:58 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:58.704607 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:58.699134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:38999)
0927 15:02:58.699405 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:58.699417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:58.699510 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:58.699759 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:58.699761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:58.699933 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:58.700049 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:58.700054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.700895 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.700898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:58.701581 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:58.701592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.702166 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.702169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:58.702256 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:58.702775 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:58.702787 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:58.704478 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
W20260927 15:02:58.704842 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:58.699233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38999 (local address 127.0.98.62:43707)
0927 15:02:58.699364 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:58.699367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:58.699516 (+ 149us) server_negotiation.cc:407] Connection header received
0927 15:02:58.699586 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:58.699588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:58.699638 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:58.699717 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:58.700189 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.700798 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.701700 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:58.702053 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:58.702157 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:58.704615 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38999: BlockingRecv error: recv got EOF from 127.0.98.3:38999 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 15:02:59 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:59 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:59.339337 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:59.333883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:36327)
0927 15:02:59.334141 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:59.334155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:59.334255 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:59.334486 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:59.334488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:59.334662 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:59.334788 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:59.334794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.335596 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.335599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:59.336248 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:59.336257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.336710 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.336713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:59.336771 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:59.337242 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:59.337252 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:59.339226 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":66,"threads_started":1}
W20260927 15:02:59.339516 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:59.333900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36327 (local address 127.0.98.62:43707)
0927 15:02:59.334174 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:59.334178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:59.334206 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:02:59.334332 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:59.334334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:59.334377 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:59.334456 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:59.334912 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.335501 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.336347 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.336618 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.336700 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:59.339351 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36327: BlockingRecv error: recv got EOF from 127.0.98.1:36327 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
Sep 27 15:02:59 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:59 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:59.519171 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:59.514019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:37943)
0927 15:02:59.514290 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:59.514310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:59.514421 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:59.514661 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:59.514663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:59.514836 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:59.514931 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:59.514936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.515719 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.515722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:59.516268 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:59.516275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.516738 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.516741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:59.516800 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:59.517299 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:59.517310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:59.519058 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
W20260927 15:02:59.519320 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:59.514160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37943 (local address 127.0.98.62:43707)
0927 15:02:59.514309 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:59.514312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:59.514364 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:02:59.514511 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:59.514513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:59.514558 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:59.514626 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:59.515050 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.515609 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.516389 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.516631 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.516723 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:59.519201 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37943: BlockingRecv error: recv got EOF from 127.0.98.2:37943 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:02:59 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:02:59 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:02:59.710615 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:02:59.705345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:50329)
0927 15:02:59.705679 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 15:02:59.705701 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:02:59.705838 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:02:59.706147 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:02:59.706149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:02:59.706323 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:02:59.706414 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:59.706419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.707140 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.707142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:59.707721 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:02:59.707728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.708160 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.708162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:02:59.708219 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:59.708729 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:02:59.708740 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:02:59.710499 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":102,"threads_started":1}
W20260927 15:02:59.710762 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:02:59.705479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50329 (local address 127.0.98.62:43707)
0927 15:02:59.705607 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:02:59.705610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:02:59.705817 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:02:59.705963 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:02:59.705965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:02:59.706010 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:02:59.706083 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:02:59.706522 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.707045 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.707831 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:02:59.708062 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:02:59.708147 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:02:59.710642 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50329: BlockingRecv error: recv got EOF from 127.0.98.3:50329 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:03:00 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:00 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:00.346133 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:00.340138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:50751)
0927 15:03:00.340520 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:00.340533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:00.340672 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:00.340944 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:00.340947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:00.341156 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:00.341244 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:00.341250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.342054 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.342061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:00.342629 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:00.342637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.343101 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.343104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:00.343190 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:00.343704 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:00.343715 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:00.346013 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":79,"threads_started":1}
W20260927 15:03:00.346297 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:00.340189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50751 (local address 127.0.98.62:43707)
0927 15:03:00.340449 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:00.340454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:00.340612 (+ 158us) server_negotiation.cc:407] Connection header received
0927 15:03:00.340772 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:00.340776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:00.340827 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:00.340924 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:00.341363 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.341950 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.342749 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.342980 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.343099 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:00.346143 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50751: BlockingRecv error: recv got EOF from 127.0.98.1:50751 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
Sep 27 15:03:00 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:00 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:00.527223 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:00.520152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:36199)
0927 15:03:00.520454 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:00.520468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:00.520575 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:00.520876 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:00.520878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:00.521067 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:00.521161 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:00.521166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.522465 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.522469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:00.523479 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:00.523493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.524248 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.524253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:00.524350 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:00.524964 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:00.524976 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:00.527093 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
W20260927 15:03:00.527551 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:00.520253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36199 (local address 127.0.98.62:43707)
0927 15:03:00.520422 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:00.520428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:00.520557 (+ 129us) server_negotiation.cc:407] Connection header received
0927 15:03:00.520672 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:00.520677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:00.520743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:00.520853 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:00.521282 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.522307 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.523618 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.524089 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.524216 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:00.527289 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36199: BlockingRecv error: recv got EOF from 127.0.98.2:36199 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:03:00 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:00 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:00.717718 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:00.711421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:44655)
0927 15:03:00.711764 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:00.711779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:00.711929 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:00.712271 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:00.712275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:00.712495 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:00.712593 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:00.712600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.713614 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.713619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:00.714262 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:00.714271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.714774 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.714778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:00.714884 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:00.715544 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:00.715556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:00.717528 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":82,"threads_started":1}
W20260927 15:03:00.717811 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:00.711534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44655 (local address 127.0.98.62:43707)
0927 15:03:00.711718 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:00.711725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:00.711855 (+ 130us) server_negotiation.cc:407] Connection header received
0927 15:03:00.712042 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:00.712046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:00.712108 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:00.712204 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:00.712726 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.713496 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.714401 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:00.714649 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:00.714773 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:00.717660 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44655: BlockingRecv error: recv got EOF from 127.0.98.3:44655 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 15:03:01 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:01 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:01.357445 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:01.346896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:56759)
0927 15:03:01.347178 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:01.347193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:01.347292 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:01.347610 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:01.347612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:01.347801 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:01.347926 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:01.347931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.348803 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.348807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:01.349465 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:01.349482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.349954 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.349957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:01.350037 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:01.355122 (+ 5085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:01.355143 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:01.357269 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
W20260927 15:03:01.357583 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:01.347091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56759 (local address 127.0.98.62:43707)
0927 15:03:01.347319 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:01.347322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:01.347335 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:03:01.347429 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:01.347431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:01.347472 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:01.347573 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:01.348035 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.348637 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.349584 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.349826 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.349956 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:01.357412 (+ 7456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56759: BlockingRecv error: recv got EOF from 127.0.98.1:56759 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
Sep 27 15:03:01 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:01 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:01.534101 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:01.528047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:58917)
0927 15:03:01.528324 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:01.528343 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:01.528488 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:01.528729 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:01.528731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:01.528985 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:01.529088 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:01.529094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.530093 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.530096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:01.530763 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:01.530776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.531280 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.531283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:01.531351 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:01.531999 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:01.532017 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:01.533988 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
W20260927 15:03:01.534415 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:01.528142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58917 (local address 127.0.98.62:43707)
0927 15:03:01.528278 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:01.528282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:01.528478 (+ 196us) server_negotiation.cc:407] Connection header received
0927 15:03:01.528581 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:01.528583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:01.528630 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:01.528695 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:01.529198 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.529955 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.530930 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.531155 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.531274 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:01.534148 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58917: BlockingRecv error: recv got EOF from 127.0.98.2:58917 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 15:03:01 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:01 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:01.723829 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:01.718582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:51827)
0927 15:03:01.718896 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:01.718910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:01.719001 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:01.719221 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:01.719223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:01.719393 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:01.719513 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:01.719518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.720212 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.720215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:01.720777 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:01.720785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.721195 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.721198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:01.721257 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:01.721711 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:01.721728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:01.723717 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
W20260927 15:03:01.723953 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:01.718617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51827 (local address 127.0.98.62:43707)
0927 15:03:01.718783 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:01.718788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:01.718960 (+ 172us) server_negotiation.cc:407] Connection header received
0927 15:03:01.719080 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:01.719082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:01.719127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:01.719185 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:01.719608 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.720122 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.720874 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:01.721103 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:01.721178 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:01.723840 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51827: BlockingRecv error: recv got EOF from 127.0.98.3:51827 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:03:02 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:02 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:02.363947 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:02.358383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:42801)
0927 15:03:02.358672 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:02.358686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:02.358775 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:02.359041 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:02.359043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:02.359215 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:02.359311 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:02.359316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.360073 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.360075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:02.360622 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:02.360629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.361099 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.361102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:02.361161 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:02.361737 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:02.361748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:02.363828 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
W20260927 15:03:02.364145 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:02.358410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42801 (local address 127.0.98.62:43707)
0927 15:03:02.358712 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:02.358716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:02.358732 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:03:02.358878 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:02.358880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:02.358926 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:02.359012 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:02.359419 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.359970 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.360740 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.360990 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.361078 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:02.363994 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42801: BlockingRecv error: recv got EOF from 127.0.98.1:42801 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
Sep 27 15:03:02 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:02 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:02.540552 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:02.534965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:55563)
0927 15:03:02.535280 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:02.535293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:02.535388 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:02.535649 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:02.535651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:02.535837 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:02.535930 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:02.535935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.536848 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.536850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:02.537455 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:02.537463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.538004 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.538007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:02.538063 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:02.538557 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:02.538568 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:02.540440 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":146,"threads_started":1}
W20260927 15:03:02.540771 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:02.534969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55563 (local address 127.0.98.62:43707)
0927 15:03:02.535184 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:02.535189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:02.535363 (+ 174us) server_negotiation.cc:407] Connection header received
0927 15:03:02.535482 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:02.535485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:02.535533 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:02.535600 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:02.536125 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.536749 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.537574 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.537890 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.537984 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:02.540627 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55563: BlockingRecv error: recv got EOF from 127.0.98.2:55563 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 15:03:02 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:02 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:02.729903 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:02.724560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:45161)
0927 15:03:02.724840 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:02.724853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:02.724947 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:02.725207 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:02.725209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:02.725382 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:02.725472 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:02.725478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.726190 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.726192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:02.726746 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:02.726752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.727144 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.727146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:02.727204 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:02.727675 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:02.727686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:02.729725 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":116,"threads_started":1}
W20260927 15:03:02.729959 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:02.724619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45161 (local address 127.0.98.62:43707)
0927 15:03:02.724760 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:02.724765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:02.724924 (+ 159us) server_negotiation.cc:407] Connection header received
0927 15:03:02.725047 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:02.725058 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:02.725104 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:02.725173 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:02.725570 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.726101 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.726841 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:02.727053 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:02.727129 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:02.729869 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45161: BlockingRecv error: recv got EOF from 127.0.98.3:45161 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:03:03 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:03 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:03.370028 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:03.364803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:47325)
0927 15:03:03.365092 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:03.365106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:03.365192 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:03.365390 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:03.365392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:03.365566 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:03.365655 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:03.365660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.366386 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.366389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:03.366952 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:03.366960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.367399 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.367403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:03.367476 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:03.368059 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:03.368070 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:03.369917 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
W20260927 15:03:03.370167 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:03.364904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47325 (local address 127.0.98.62:43707)
0927 15:03:03.365157 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:03.365160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:03.365173 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:03:03.365260 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:03.365262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:03.365299 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:03.365372 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:03.365752 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.366281 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.367060 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.367272 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.367347 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:03.370038 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47325: BlockingRecv error: recv got EOF from 127.0.98.1:47325 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":61,"threads_started":1}
Sep 27 15:03:03 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:03 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:03.547185 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:03.541508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:49331)
0927 15:03:03.541811 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:03.541824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:03.541922 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:03.542161 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:03.542163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:03.542343 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:03.542433 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:03.542438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.543186 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.543188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:03.543751 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:03.543758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.544226 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.544229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:03.544308 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:03.544972 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:03.544991 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:03.547017 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
W20260927 15:03:03.547264 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:03.541578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49331 (local address 127.0.98.62:43707)
0927 15:03:03.541732 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:03.541738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:03.541902 (+ 164us) server_negotiation.cc:407] Connection header received
0927 15:03:03.542004 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:03.542006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:03.542051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:03.542128 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:03.542578 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.543094 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.543878 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.544103 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.544193 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:03.547150 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49331: BlockingRecv error: recv got EOF from 127.0.98.2:49331 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 15:03:03 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:03 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:03.736999 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:03.730708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:39045)
0927 15:03:03.731102 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:03.731115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:03.731223 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:03.731544 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:03.731547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:03.731758 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:03.731906 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:03.731914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.732876 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.732880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:03.733732 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:03.733742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.734273 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.734277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:03.734350 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:03.734838 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:03.734850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:03.736874 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
W20260927 15:03:03.737259 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:03.730900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39045 (local address 127.0.98.62:43707)
0927 15:03:03.731021 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:03.731024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:03.731184 (+ 160us) server_negotiation.cc:407] Connection header received
0927 15:03:03.731363 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:03.731370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:03.731433 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:03.731517 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:03.732061 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.732721 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.733909 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:03.734132 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:03.734218 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:03.737036 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39045: BlockingRecv error: recv got EOF from 127.0.98.3:39045 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 15:03:04 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:04 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:04.375936 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:04.370778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:53731)
0927 15:03:04.371085 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:04.371099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:04.371199 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:04.371443 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:04.371445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:04.371623 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:04.371714 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:04.371719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.372452 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.372455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:04.373026 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:04.373034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.373468 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.373471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:04.373533 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:04.374052 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:04.374063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:04.375819 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
W20260927 15:03:04.376117 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:04.370890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53731 (local address 127.0.98.62:43707)
0927 15:03:04.371121 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:04.371124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:04.371152 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:03:04.371284 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:04.371286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:04.371327 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:04.371402 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:04.371810 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.372330 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.373134 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.373352 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.373429 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:04.375948 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53731: BlockingRecv error: recv got EOF from 127.0.98.1:53731 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
Sep 27 15:03:04 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:04 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:04.553570 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:04.547941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:54701)
0927 15:03:04.548274 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:04.548287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:04.548471 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:04.548741 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:04.548744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:04.548958 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:04.549048 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:04.549054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.549820 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.549823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:04.550405 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:04.550414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.550887 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.550890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:04.550974 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:04.551472 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:04.551483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:04.553403 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":84,"threads_started":1}
W20260927 15:03:04.553663 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:04.548064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54701 (local address 127.0.98.62:43707)
0927 15:03:04.548224 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:04.548228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:04.548406 (+ 178us) server_negotiation.cc:407] Connection header received
0927 15:03:04.548571 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:04.548574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:04.548627 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:04.548723 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:04.549165 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.549687 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.550535 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.550758 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.550874 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:04.553547 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54701: BlockingRecv error: recv got EOF from 127.0.98.2:54701 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 15:03:04 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:04 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:04.743219 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:04.737684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:59867)
0927 15:03:04.737978 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:04.738017 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:04.738116 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:04.738355 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:04.738358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:04.738530 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:04.738620 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:04.738624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.739327 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.739329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:04.739924 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:04.739931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.740319 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.740321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:04.740412 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:04.740880 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:04.740895 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:04.743018 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
W20260927 15:03:04.743234 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:04.737786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59867 (local address 127.0.98.62:43707)
0927 15:03:04.737930 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:04.737933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:04.738094 (+ 161us) server_negotiation.cc:407] Connection header received
0927 15:03:04.738204 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:04.738205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:04.738248 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:04.738315 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:04.738713 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.739239 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.740022 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:04.740229 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:04.740305 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:04.743114 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59867: BlockingRecv error: recv got EOF from 127.0.98.3:59867 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 15:03:05 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:05 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:05.382059 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:05.376758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:48599)
0927 15:03:05.377029 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:05.377042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:05.377129 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:05.377372 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:05.377374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:05.377566 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:05.377662 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:05.377667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.378515 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.378517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:05.379081 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:05.379089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.379552 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.379554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:05.379613 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:05.380139 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:05.380150 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:05.381927 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260927 15:03:05.382283 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:05.376842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48599 (local address 127.0.98.62:43707)
0927 15:03:05.377119 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:05.377123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:05.377138 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:03:05.377198 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:05.377200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:05.377244 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:05.377358 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:05.377822 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.378410 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.379207 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.379448 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.379542 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:05.382085 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48599: BlockingRecv error: recv got EOF from 127.0.98.1:48599 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
Sep 27 15:03:05 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:05 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:05.559646 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:05.554378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:45253)
0927 15:03:05.554644 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:05.554658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:05.554756 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:05.555010 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:05.555012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:05.555187 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:05.555279 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:05.555283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.556018 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.556020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:05.556570 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:05.556576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.557060 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.557063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:05.557124 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:05.557624 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:05.557635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:05.559470 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":65,"threads_started":1}
W20260927 15:03:05.559783 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:05.554485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45253 (local address 127.0.98.62:43707)
0927 15:03:05.554657 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:05.554661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:05.554738 (+ 77us) server_negotiation.cc:407] Connection header received
0927 15:03:05.554839 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:05.554841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:05.554895 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:05.554964 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:05.555394 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.555913 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.556693 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.556941 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.557033 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:05.559667 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45253: BlockingRecv error: recv got EOF from 127.0.98.2:45253 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 15:03:05 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:05 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:05.749577 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:05.744001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:40851)
0927 15:03:05.744284 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:05.744298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:05.744434 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:05.744727 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:05.744730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:05.744939 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:05.745044 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:05.745050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.745951 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.745957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:05.746536 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:05.746544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.747013 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.747017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:05.747105 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:05.747647 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:05.747657 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:05.749397 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
W20260927 15:03:05.749611 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:05.744069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40851 (local address 127.0.98.62:43707)
0927 15:03:05.744242 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:05.744246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:05.744384 (+ 138us) server_negotiation.cc:407] Connection header received
0927 15:03:05.744556 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:05.744560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:05.744615 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:05.744703 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:05.745166 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.745843 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.746656 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:05.746897 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:05.747015 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:05.749512 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40851: BlockingRecv error: recv got EOF from 127.0.98.3:40851 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 15:03:06 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:06 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:06.394481 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:06.382865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:33653)
0927 15:03:06.383169 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:06.383184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:06.383284 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:06.383597 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:06.383600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:06.383792 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:06.383892 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:06.383898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.384647 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.384649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:06.385323 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:06.385332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.385984 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.385990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:06.386074 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:06.392223 (+ 6149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:06.392237 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:06.394280 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":127,"threads_started":1}
W20260927 15:03:06.394529 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:06.382922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33653 (local address 127.0.98.62:43707)
0927 15:03:06.383165 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:06.383168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:06.383235 (+ 67us) server_negotiation.cc:407] Connection header received
0927 15:03:06.383438 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:06.383440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:06.383480 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:06.383558 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:06.383995 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.384535 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.385452 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.385836 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.386169 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:06.394409 (+ 8240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33653: BlockingRecv error: recv got EOF from 127.0.98.1:33653 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
Sep 27 15:03:06 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:06 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:06.565713 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:06.560463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:34383)
0927 15:03:06.560729 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:06.560742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:06.560840 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:06.561070 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:06.561072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:06.561248 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:06.561398 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:06.561403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.562196 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.562198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:06.562809 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:06.562817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.563231 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.563233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:06.563291 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:06.563802 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:06.563815 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:06.565601 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
W20260927 15:03:06.565894 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:06.560607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34383 (local address 127.0.98.62:43707)
0927 15:03:06.560737 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:06.560741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:06.560796 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:03:06.560919 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:06.560921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:06.560966 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:06.561033 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:06.561567 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.562106 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.562917 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.563131 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.563209 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:06.565733 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34383: BlockingRecv error: recv got EOF from 127.0.98.2:34383 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 15:03:06 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:06 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:06.755927 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:06.750428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:53735)
0927 15:03:06.750700 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:06.750712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:06.750816 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:06.751056 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:06.751058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:06.751233 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:06.751335 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:06.751341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.752068 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.752071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:06.752648 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:06.752656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.753058 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.753061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:06.753127 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:06.753614 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:06.753626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:06.755769 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
W20260927 15:03:06.759989 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:06.750499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53735 (local address 127.0.98.62:43707)
0927 15:03:06.750624 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:06.750627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:06.750787 (+ 160us) server_negotiation.cc:407] Connection header received
0927 15:03:06.750895 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:06.750897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:06.750941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:06.751018 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:06.751433 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.751953 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.752752 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:06.752961 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:06.753035 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:06.759830 (+ 6795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53735: BlockingRecv error: recv got EOF from 127.0.98.3:53735 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 15:03:07 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:07 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:07.400784 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:07.395223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:34693)
0927 15:03:07.395492 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:07.395508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:07.395596 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:07.395824 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:07.395826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:07.395996 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:07.396095 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:07.396100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.396881 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.396884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:07.397492 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:07.397501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.398038 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.398041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:07.398116 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:07.398663 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:07.398676 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:07.400653 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
W20260927 15:03:07.400935 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:07.395303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34693 (local address 127.0.98.62:43707)
0927 15:03:07.395589 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:07.395593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:07.395605 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:03:07.395665 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:07.395667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:07.395707 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:07.395782 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:07.396194 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.396751 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.397606 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.397899 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.397980 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:07.400788 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34693: BlockingRecv error: recv got EOF from 127.0.98.1:34693 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":81,"threads_started":1}
Sep 27 15:03:07 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:07 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:07.572456 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:07.566603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:60093)
0927 15:03:07.566888 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:07.566904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:07.567025 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:07.567413 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:07.567424 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:07.567706 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:07.567849 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:07.567857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.568675 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.568678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:07.569351 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:07.569360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.569850 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.569852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:07.569917 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:07.570398 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:07.570410 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:07.572311 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
W20260927 15:03:07.572577 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:07.566669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60093 (local address 127.0.98.62:43707)
0927 15:03:07.566825 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:07.566829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:07.566987 (+ 158us) server_negotiation.cc:407] Connection header received
0927 15:03:07.567136 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:07.567139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:07.567202 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:07.567289 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:07.567999 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.568558 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.569493 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.569707 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.569872 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:07.572452 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60093: BlockingRecv error: recv got EOF from 127.0.98.2:60093 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 15:03:07 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:07 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:07.762415 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:07.756717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.3:40939)
0927 15:03:07.757063 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:07.757077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:07.757223 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:07.757511 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:07.757514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:07.757718 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:07.757837 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:07.757843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.758583 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.758588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:07.759145 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:07.759152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.759605 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.759609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:07.759694 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:07.760212 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:07.760224 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:07.762216 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":91,"threads_started":1}
W20260927 15:03:07.762640 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:07.756837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40939 (local address 127.0.98.62:43707)
0927 15:03:07.757027 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:07.757031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:07.757163 (+ 132us) server_negotiation.cc:407] Connection header received
0927 15:03:07.757337 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:07.757340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:07.757400 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:07.757489 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:07.757950 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.758475 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.759258 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:07.759488 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:07.759597 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:07.762381 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40939: BlockingRecv error: recv got EOF from 127.0.98.3:40939 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 27 15:03:08 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:08 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:08.406971 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:08.401843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.1:39711)
0927 15:03:08.402108 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:08.402122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:08.402207 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:08.402473 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:08.402475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:08.402648 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:08.402740 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:08.402746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.403516 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.403519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:08.404064 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:08.404071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.404528 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.404531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:08.404590 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:08.405069 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:08.405080 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:08.406861 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
W20260927 15:03:08.407183 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:08.401900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39711 (local address 127.0.98.62:43707)
0927 15:03:08.402177 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:08.402180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:08.402194 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:03:08.402296 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:08.402299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:08.402346 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:08.402439 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:08.402852 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.403416 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.404178 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.404429 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.404514 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:08.407030 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39711: BlockingRecv error: recv got EOF from 127.0.98.1:39711 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
Sep 27 15:03:08 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:03:08 dist-test-slave-ml7t krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:03:08.579568 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:08.573480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43707 (local address 127.0.98.2:51333)
0927 15:03:08.573754 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:08.573822 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:08.573933 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:08.574278 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:08.574280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:08.574452 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:08.574543 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:08.574575 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.575582 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.575588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:08.576233 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:08.576244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.576729 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.576733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:08.576832 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:08.577428 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:08.577440 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:08.579347 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43707: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
W20260927 15:03:08.579608 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:08.573640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51333 (local address 127.0.98.62:43707)
0927 15:03:08.573862 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:08.573867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:08.573891 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:03:08.574087 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:08.574090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:08.574153 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:08.574246 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:08.574701 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.575452 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.576365 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.576606 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.576725 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:08.579489 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51333: BlockingRecv error: recv got EOF from 127.0.98.2:51333 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:03:08.666781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9131
I20260927 15:03:08.672288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9283
I20260927 15:03:08.677428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9415
I20260927 15:03:08.682508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9061
2026-09-27T15:03:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18066 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T15:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:03:08Z Disabled control of system clock
I20260927 15:03:08.712266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33733
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:03:08.820104 9611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:08.820384 9611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:08.820420 9611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:08.824484 9611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:03:08.824548 9611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:08.824563 9611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:08.824581 9611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:03:08.824596 9611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:03:08.829505 9611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:08.830664 9611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:08.832077 9611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:08.840394 9619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:08.840423 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:08.840408 9617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:08.840656 9611 server_base.cc:1034] running on GCE node
I20260927 15:03:08.841308 9611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:08.842442 9611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:08.843627 9611 hybrid_clock.cc:648] HybridClock initialized: now 1790521388843611 us; error 32 us; skew 500 ppm
I20260927 15:03:08.845512 9611 webserver.cc:460] Webserver started at http://127.0.98.62:42815/ using document root <none> and password file <none>
I20260927 15:03:08.846134 9611 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:08.846220 9611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:08.846441 9611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:08.848487 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "04fce2324d16473ea19be1dff6410e38"
format_stamp: "Formatted at 2026-09-27 15:03:08 on dist-test-slave-ml7t"
server_key: "340ead6dac70d368bd97b4f732d7ada8"
server_key_iv: "ecc5f66802454db9f30c1230b0b71182"
server_key_version: "encryptionkey@0"
I20260927 15:03:08.849061 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "04fce2324d16473ea19be1dff6410e38"
format_stamp: "Formatted at 2026-09-27 15:03:08 on dist-test-slave-ml7t"
server_key: "340ead6dac70d368bd97b4f732d7ada8"
server_key_iv: "ecc5f66802454db9f30c1230b0b71182"
server_key_version: "encryptionkey@0"
I20260927 15:03:08.852739 9611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:03:08.855229 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:08.856489 9611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:03:08.856624 9611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "04fce2324d16473ea19be1dff6410e38"
format_stamp: "Formatted at 2026-09-27 15:03:08 on dist-test-slave-ml7t"
server_key: "340ead6dac70d368bd97b4f732d7ada8"
server_key_iv: "ecc5f66802454db9f30c1230b0b71182"
server_key_version: "encryptionkey@0"
I20260927 15:03:08.856737 9611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:08.884039 9611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:08.884752 9611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:08.884944 9611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:08.892205 9611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33733
I20260927 15:03:08.892213 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33733 every 8 connection(s)
I20260927 15:03:08.893154 9611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:03:08.896209 9678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:08.898048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9611
I20260927 15:03:08.898156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:03:08.898461 392 external_mini_cluster.cc:1428] Setting key 1e248747865af94297bd9edd18fd8782
I20260927 15:03:08.902820 9678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38: Bootstrap starting.
I20260927 15:03:08.905071 9678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:08.905834 9678 log.cc:824] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:08.907511 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:08.899932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.98.62:33733)
0927 15:03:08.900566 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:08.900581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:08.900616 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:03:08.901263 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:08.901280 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:08.901655 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:08.902049 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:08.902468 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.903508 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.904361 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:08.904814 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:08.905000 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:08.905854 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:08.905859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:08.905871 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:08.906295 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:08.906348 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:08.906367 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:08.906478 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:08.906604 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:03:08.906848 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260927 15:03:08.907727 9678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38: No bootstrap required, opened a new log
I20260927 15:03:08.911567 9678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fce2324d16473ea19be1dff6410e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33733 } }
I20260927 15:03:08.911799 9678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:08.911885 9678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fce2324d16473ea19be1dff6410e38, State: Initialized, Role: FOLLOWER
I20260927 15:03:08.912286 9678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fce2324d16473ea19be1dff6410e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33733 } }
I20260927 15:03:08.912432 9678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:03:08.912518 9678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:03:08.912611 9678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:08.913472 9678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fce2324d16473ea19be1dff6410e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33733 } }
I20260927 15:03:08.913838 9678 leader_election.cc:302] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04fce2324d16473ea19be1dff6410e38; no voters:
I20260927 15:03:08.914157 9678 leader_election.cc:288] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:03:08.914238 9683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:03:08.914496 9683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [term 1 LEADER]: Becoming Leader. State: Replica: 04fce2324d16473ea19be1dff6410e38, State: Running, Role: LEADER
I20260927 15:03:08.914826 9683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fce2324d16473ea19be1dff6410e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33733 } }
I20260927 15:03:08.915467 9678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:03:08.916524 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04fce2324d16473ea19be1dff6410e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fce2324d16473ea19be1dff6410e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33733 } } }
I20260927 15:03:08.916551 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04fce2324d16473ea19be1dff6410e38. Latest consensus state: current_term: 1 leader_uuid: "04fce2324d16473ea19be1dff6410e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fce2324d16473ea19be1dff6410e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33733 } } }
I20260927 15:03:08.916654 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [sys.catalog]: This master's current role is: LEADER
I20260927 15:03:08.916653 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38 [sys.catalog]: This master's current role is: LEADER
I20260927 15:03:08.917006 9692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:03:08.920166 9692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:03:08.926276 9692 catalog_manager.cc:1347] Generated new cluster ID: 86c428cd697745929b8371dcf21729ea
I20260927 15:03:08.926348 9692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:03:08.939262 9692 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:03:08.940467 9692 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:03:08.945448 9692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04fce2324d16473ea19be1dff6410e38: Generated new TSK 0
I20260927 15:03:08.946251 9692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:03:08.957170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33733
--builtin_ntp_servers=127.0.98.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 15:03:09.068019 9702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:09.068308 9702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:09.068389 9702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:09.072784 9702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:09.072860 9702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:09.072945 9702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:03:09.077996 9702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:09.079087 9702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:09.080449 9702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:09.090653 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:09.090665 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.090866 9702 server_base.cc:1034] running on GCE node
W20260927 15:03:09.090937 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.091389 9702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:09.092295 9702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:09.093490 9702 hybrid_clock.cc:648] HybridClock initialized: now 1790521389093470 us; error 44 us; skew 500 ppm
I20260927 15:03:09.095544 9702 webserver.cc:460] Webserver started at http://127.0.98.1:45607/ using document root <none> and password file <none>
I20260927 15:03:09.096161 9702 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:09.096225 9702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:09.096455 9702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:09.098604 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "7501aed93eb4429c8af1ba728221be9e"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "d9ab7f437ca2bec37c0d41d90a4f60ce"
server_key_iv: "222649b3b144fb60b298c161d0b18bbd"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.099155 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7501aed93eb4429c8af1ba728221be9e"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "d9ab7f437ca2bec37c0d41d90a4f60ce"
server_key_iv: "222649b3b144fb60b298c161d0b18bbd"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.102902 9702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:03:09.105499 9716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.106810 9702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:03:09.106935 9702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "7501aed93eb4429c8af1ba728221be9e"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "d9ab7f437ca2bec37c0d41d90a4f60ce"
server_key_iv: "222649b3b144fb60b298c161d0b18bbd"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.107044 9702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:09.133252 9702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:09.134069 9702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:09.134268 9702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:09.134867 9702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:09.135848 9702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:09.135918 9702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.136000 9702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:09.136052 9702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.144985 9702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35553
I20260927 15:03:09.145058 9829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35553 every 8 connection(s)
I20260927 15:03:09.146036 9702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:03:09.153918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9702
I20260927 15:03:09.154050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:03:09.154392 392 external_mini_cluster.cc:1428] Setting key f3815569568894e956276bf320654ae4
I20260927 15:03:09.155575 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:09.148027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54109 (local address 127.0.98.62:33733)
0927 15:03:09.148171 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:09.148175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:09.149257 (+ 1082us) server_negotiation.cc:407] Connection header received
0927 15:03:09.150719 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:09.150722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:09.150768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:09.150845 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:09.151705 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.152241 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.153461 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.153677 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.153759 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:09.155087 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:09.155090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:09.155092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:09.155102 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:09.155141 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:09.155150 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:09.155224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:09.155387 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 15:03:09.155443 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:03:09.156481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33733
--builtin_ntp_servers=127.0.98.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:03:09.156540 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:09.148446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33733 (local address 127.0.98.1:54109)
0927 15:03:09.149109 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:09.149153 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:09.150500 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:09.151008 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:09.151014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:09.151098 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:09.151582 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:09.151594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.152378 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.152381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:09.153339 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:09.153349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.153879 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.153884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:09.154114 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:09.154786 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:03:09.154788 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:09.154842 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:03:09.154906 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:09.155236 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:09.155243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:09.155248 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:03:09.155357 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:03:09.155578 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260927 15:03:09.157810 9830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33733
I20260927 15:03:09.158075 9830 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:09.158593 9830 heartbeater.cc:511] Master 127.0.98.62:33733 requested a full tablet report, sending...
I20260927 15:03:09.160216 9642 ts_manager.cc:194] Registered new tserver with Master: 7501aed93eb4429c8af1ba728221be9e (127.0.98.1:35553)
I20260927 15:03:09.161983 9642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54109
W20260927 15:03:09.267215 9833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:09.267477 9833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:09.267513 9833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:09.271462 9833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:09.271518 9833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:09.271571 9833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:03:09.276618 9833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:09.277671 9833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:09.279129 9833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:09.289062 9839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:09.289086 9838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:09.289069 9841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.289238 9833 server_base.cc:1034] running on GCE node
I20260927 15:03:09.289714 9833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:09.290428 9833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:09.291590 9833 hybrid_clock.cc:648] HybridClock initialized: now 1790521389291557 us; error 50 us; skew 500 ppm
I20260927 15:03:09.293396 9833 webserver.cc:460] Webserver started at http://127.0.98.2:40379/ using document root <none> and password file <none>
I20260927 15:03:09.294055 9833 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:09.294119 9833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:09.294325 9833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:09.296346 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7c933c49cca4d99acfb149a9caaad4e"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "7a3dd1d0beacc3c0d7335b9f3e70b848"
server_key_iv: "4d622e21b3e31155ee5178e25fb29ca5"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.296918 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7c933c49cca4d99acfb149a9caaad4e"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "7a3dd1d0beacc3c0d7335b9f3e70b848"
server_key_iv: "4d622e21b3e31155ee5178e25fb29ca5"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.300578 9833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 15:03:09.303323 9847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.304556 9833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:03:09.304670 9833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "b7c933c49cca4d99acfb149a9caaad4e"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "7a3dd1d0beacc3c0d7335b9f3e70b848"
server_key_iv: "4d622e21b3e31155ee5178e25fb29ca5"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.304780 9833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:09.323508 9833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:09.324175 9833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:09.324366 9833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:09.324939 9833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:09.325907 9833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:09.325953 9833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.326015 9833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:09.326048 9833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.335038 9833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42801
I20260927 15:03:09.335060 9960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42801 every 8 connection(s)
I20260927 15:03:09.335915 9833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:03:09.343253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9833
I20260927 15:03:09.343374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:03:09.343703 392 external_mini_cluster.cc:1428] Setting key 5017fbfa9486e9eafd1971b5145a9262
I20260927 15:03:09.345492 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:09.337912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58683 (local address 127.0.98.62:33733)
0927 15:03:09.338040 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:09.338044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:09.339132 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 15:03:09.340447 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:09.340450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:09.340497 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:09.340577 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:09.341450 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.342015 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.343157 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.343505 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.343642 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:09.344851 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:09.344855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:09.344858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:09.344870 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:09.344894 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:09.344928 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:09.345030 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:09.345282 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 15:03:09.345348 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:03:09.346197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33733
--builtin_ntp_servers=127.0.98.20:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:03:09.346632 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:09.338314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33733 (local address 127.0.98.2:58683)
0927 15:03:09.338950 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:09.338990 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:09.340241 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:09.340731 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:09.340737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:09.340818 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:09.341315 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:09.341328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.342129 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.342132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:09.343011 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:09.343020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.343666 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.343670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:09.343868 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:09.344556 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:03:09.344558 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:09.344617 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:03:09.344686 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:09.345066 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:09.345094 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:09.345102 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:03:09.345244 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:03:09.345542 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260927 15:03:09.347862 9961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33733
I20260927 15:03:09.348135 9961 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:09.348769 9961 heartbeater.cc:511] Master 127.0.98.62:33733 requested a full tablet report, sending...
I20260927 15:03:09.349911 9642 ts_manager.cc:194] Registered new tserver with Master: b7c933c49cca4d99acfb149a9caaad4e (127.0.98.2:42801)
I20260927 15:03:09.350544 9642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58683
W20260927 15:03:09.459986 9964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:09.460295 9964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:09.460355 9964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:09.464536 9964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:09.464598 9964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:09.464690 9964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:03:09.469480 9964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:09.470590 9964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:09.471943 9964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:09.481508 9972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:09.481626 9969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.481668 9964 server_base.cc:1034] running on GCE node
W20260927 15:03:09.481909 9970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.482352 9964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:09.482969 9964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:09.484153 9964 hybrid_clock.cc:648] HybridClock initialized: now 1790521389484144 us; error 44 us; skew 500 ppm
I20260927 15:03:09.486016 9964 webserver.cc:460] Webserver started at http://127.0.98.3:38447/ using document root <none> and password file <none>
I20260927 15:03:09.486618 9964 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:09.486708 9964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:09.486897 9964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:09.488797 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa66f7b310cc42888f55044bd2bb7c98"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "e40a486bcc1c82a501d4910e6ee8ba82"
server_key_iv: "87c89e5c0ded6c6bf0ee085d0aefda2e"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.489295 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa66f7b310cc42888f55044bd2bb7c98"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "e40a486bcc1c82a501d4910e6ee8ba82"
server_key_iv: "87c89e5c0ded6c6bf0ee085d0aefda2e"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.492780 9964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 15:03:09.495126 9978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.496297 9964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:03:09.496429 9964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "aa66f7b310cc42888f55044bd2bb7c98"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "e40a486bcc1c82a501d4910e6ee8ba82"
server_key_iv: "87c89e5c0ded6c6bf0ee085d0aefda2e"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.496547 9964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:09.515933 9964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:09.516741 9964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:09.516942 9964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:09.517516 9964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:09.518476 9964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:09.518545 9964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.518611 9964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:09.518666 9964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.527513 9964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41579
I20260927 15:03:09.527532 10091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41579 every 8 connection(s)
I20260927 15:03:09.528447 9964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:03:09.533070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 9964
I20260927 15:03:09.533195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:03:09.533537 392 external_mini_cluster.cc:1428] Setting key ce206241e636a88f2bfebb2444c290a8
I20260927 15:03:09.538318 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:09.530366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57289 (local address 127.0.98.62:33733)
0927 15:03:09.530519 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:09.530522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:09.531675 (+ 1153us) server_negotiation.cc:407] Connection header received
0927 15:03:09.533086 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:09.533093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:09.533159 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:09.533282 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:09.534447 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.535198 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.536483 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.536702 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.536801 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:09.537898 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:09.537901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:09.537903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:09.537915 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:09.537935 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:09.537944 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:09.538012 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:09.538143 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 15:03:09.538193 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:03:09.539173 10094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:09.530800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33733 (local address 127.0.98.3:57289)
0927 15:03:09.531521 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:09.531563 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:09.532862 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:09.533470 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:09.533479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:09.533566 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:09.534280 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:09.534300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.535330 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.535333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:09.536362 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:09.536371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:09.536831 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:09.536834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:09.536984 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:09.537624 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:03:09.537626 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:09.537664 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:03:09.537726 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:09.538023 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:09.538029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:09.538033 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:03:09.538126 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:03:09.538333 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":111,"threads_started":1}
I20260927 15:03:09.540251 10092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33733
I20260927 15:03:09.540516 10092 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:09.541007 10092 heartbeater.cc:511] Master 127.0.98.62:33733 requested a full tablet report, sending...
I20260927 15:03:09.542057 9642 ts_manager.cc:194] Registered new tserver with Master: aa66f7b310cc42888f55044bd2bb7c98 (127.0.98.3:41579)
I20260927 15:03:09.542587 9642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57289
I20260927 15:03:09.547400 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:03:09.550138 9681 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:09.549268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.0.98.62:33733)
0927 15:03:09.549430 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:09.549433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:09.549519 (+ 86us) server_negotiation.cc:407] Connection header received
0927 15:03:09.549613 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:09.549615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:09.549651 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:09.549714 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:09.550032 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33704: BlockingRecv error: recv got EOF from 127.0.0.1:33704 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 15:03:09.550266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9702
I20260927 15:03:09.556136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9833
I20260927 15:03:09.561443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9964
I20260927 15:03:09.566457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 9611
2026-09-27T15:03:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (883 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T15:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:03:09Z Disabled control of system clock
I20260927 15:03:09.595819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42229
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:03:09.714022 10103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:09.714286 10103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:09.714347 10103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:09.718374 10103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:03:09.718446 10103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:09.718464 10103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:09.718483 10103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:03:09.718502 10103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:03:09.723500 10103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:09.724604 10103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:09.726086 10103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:09.734539 10108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:09.734615 10111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:09.734541 10109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.734652 10103 server_base.cc:1034] running on GCE node
I20260927 15:03:09.735311 10103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:09.736402 10103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:09.737569 10103 hybrid_clock.cc:648] HybridClock initialized: now 1790521389737554 us; error 26 us; skew 500 ppm
I20260927 15:03:09.739512 10103 webserver.cc:460] Webserver started at http://127.0.98.62:37755/ using document root <none> and password file <none>
I20260927 15:03:09.740072 10103 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:09.740134 10103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:09.740347 10103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:09.742476 10103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "25ca676e30eb45798125236038f6d5e6"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "9803d7e9b7a1de6cabcb3b1a023947e7"
server_key_iv: "143a2c9ed8056f804004e57170a00120"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.743023 10103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "25ca676e30eb45798125236038f6d5e6"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "9803d7e9b7a1de6cabcb3b1a023947e7"
server_key_iv: "143a2c9ed8056f804004e57170a00120"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.746740 10103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:03:09.749047 10117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.750315 10103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:03:09.750473 10103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "25ca676e30eb45798125236038f6d5e6"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "9803d7e9b7a1de6cabcb3b1a023947e7"
server_key_iv: "143a2c9ed8056f804004e57170a00120"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.750586 10103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:09.754959 10103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:09.755535 10103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:09.755712 10103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:09.762151 10103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42229
I20260927 15:03:09.762166 10169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42229 every 8 connection(s)
I20260927 15:03:09.763056 10103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:03:09.766114 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:09.771322 10170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6: Bootstrap starting.
I20260927 15:03:09.771615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10103
I20260927 15:03:09.771726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:03:09.772025 392 external_mini_cluster.cc:1428] Setting key b229fdc39d8bf44681e1113028136dcd
I20260927 15:03:09.774137 10170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:09.774896 10170 log.cc:824] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:09.776906 10170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6: No bootstrap required, opened a new log
I20260927 15:03:09.777529 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:09.773356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.0.98.62:42229)
0927 15:03:09.774060 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:09.774076 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:09.774111 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:03:09.774764 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:09.774779 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:09.775180 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:09.775516 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:09.776291 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:09.776295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:09.776307 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:09.776317 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:09.776371 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:09.776386 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:09.776501 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:09.776611 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 15:03:09.776842 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":130,"threads_started":1}
I20260927 15:03:09.780786 10170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca676e30eb45798125236038f6d5e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260927 15:03:09.780994 10170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:09.781078 10170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ca676e30eb45798125236038f6d5e6, State: Initialized, Role: FOLLOWER
I20260927 15:03:09.781456 10170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca676e30eb45798125236038f6d5e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260927 15:03:09.781569 10170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:03:09.781651 10170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:03:09.781749 10170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:09.782683 10170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca676e30eb45798125236038f6d5e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260927 15:03:09.783028 10170 leader_election.cc:302] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ca676e30eb45798125236038f6d5e6; no voters:
I20260927 15:03:09.783322 10170 leader_election.cc:288] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:03:09.783604 10175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:03:09.783838 10175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [term 1 LEADER]: Becoming Leader. State: Replica: 25ca676e30eb45798125236038f6d5e6, State: Running, Role: LEADER
I20260927 15:03:09.784148 10175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca676e30eb45798125236038f6d5e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } }
I20260927 15:03:09.784533 10170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:03:09.785871 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ca676e30eb45798125236038f6d5e6. Latest consensus state: current_term: 1 leader_uuid: "25ca676e30eb45798125236038f6d5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca676e30eb45798125236038f6d5e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } } }
I20260927 15:03:09.786005 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:03:09.786324 10176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ca676e30eb45798125236038f6d5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca676e30eb45798125236038f6d5e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42229 } } }
I20260927 15:03:09.786410 10176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6 [sys.catalog]: This master's current role is: LEADER
W20260927 15:03:09.789052 10190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:03:09.789153 10190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:03:09.789234 10185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:03:09.790038 10185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:03:09.795537 10185 catalog_manager.cc:1347] Generated new cluster ID: 3f29ad541f564fbbb9db79b032937441
I20260927 15:03:09.795620 10185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:03:09.834098 10185 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:03:09.834471 10185 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:03:09.842057 10185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ca676e30eb45798125236038f6d5e6: Generated new TSK 0
I20260927 15:03:09.842819 10185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:03:09.847249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42229
--builtin_ntp_servers=127.0.98.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:03:09.958753 10194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:09.959014 10194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:09.959054 10194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:09.963186 10194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:09.963238 10194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:09.963320 10194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:03:09.967998 10194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:09.969059 10194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:09.970471 10194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:09.979528 10202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:09.979501 10200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.979655 10194 server_base.cc:1034] running on GCE node
W20260927 15:03:09.979517 10199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:09.980230 10194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:09.980808 10194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:09.981988 10194 hybrid_clock.cc:648] HybridClock initialized: now 1790521389981976 us; error 24 us; skew 500 ppm
I20260927 15:03:09.983731 10194 webserver.cc:460] Webserver started at http://127.0.98.1:39221/ using document root <none> and password file <none>
I20260927 15:03:09.984273 10194 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:09.984362 10194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:09.984606 10194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:09.986811 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "49f0a91bf08c498f9bcc2bb2aa99f08b"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "0abb53b582a0916d2f5b56366b66c869"
server_key_iv: "0d3b2345cb17282b3872c9f623cacf8c"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.987356 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "49f0a91bf08c498f9bcc2bb2aa99f08b"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "0abb53b582a0916d2f5b56366b66c869"
server_key_iv: "0d3b2345cb17282b3872c9f623cacf8c"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.990936 10194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:03:09.993409 10208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:09.994678 10194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:03:09.994808 10194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "49f0a91bf08c498f9bcc2bb2aa99f08b"
format_stamp: "Formatted at 2026-09-27 15:03:09 on dist-test-slave-ml7t"
server_key: "0abb53b582a0916d2f5b56366b66c869"
server_key_iv: "0d3b2345cb17282b3872c9f623cacf8c"
server_key_version: "encryptionkey@0"
I20260927 15:03:09.994920 10194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:09.999778 10194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:10.000437 10194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:10.000627 10194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:10.001195 10194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:10.002211 10194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:10.002290 10194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.002384 10194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:10.002503 10194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.012264 10194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45893
I20260927 15:03:10.012290 10321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45893 every 8 connection(s)
I20260927 15:03:10.013202 10194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:03:10.013409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10194
I20260927 15:03:10.013514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:03:10.013882 392 external_mini_cluster.cc:1428] Setting key 2091799fa88abb4705717c1c414ce243
I20260927 15:03:10.015861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42229
--builtin_ntp_servers=127.0.98.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:03:10.019860 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:10.015176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37683 (local address 127.0.98.62:42229)
0927 15:03:10.015320 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:10.015324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:10.016476 (+ 1152us) server_negotiation.cc:407] Connection header received
0927 15:03:10.018135 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:10.018138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:10.018171 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:10.018231 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:10.019412 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:10.019415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:10.019417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:10.019427 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:10.019447 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:10.019454 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:10.019535 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:10.019724 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 15:03:10.019761 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:03:10.021126 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:10.015625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42229 (local address 127.0.98.1:37683)
0927 15:03:10.016309 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:10.016360 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:10.017906 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:10.018399 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:10.018411 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:10.018536 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:10.019108 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:03:10.019112 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:10.019166 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:03:10.019254 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:10.019557 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:10.019569 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:10.019579 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 15:03:10.019766 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 15:03:10.020028 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":129,"threads_started":1}
I20260927 15:03:10.022341 10322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42229
I20260927 15:03:10.022596 10322 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:10.023084 10322 heartbeater.cc:511] Master 127.0.98.62:42229 requested a full tablet report, sending...
I20260927 15:03:10.024571 10134 ts_manager.cc:194] Registered new tserver with Master: 49f0a91bf08c498f9bcc2bb2aa99f08b (127.0.98.1:45893)
W20260927 15:03:10.125761 10325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:10.126048 10325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:10.126132 10325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:10.130120 10325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:10.130179 10325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:10.130285 10325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:03:10.135164 10325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:10.136284 10325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:10.137665 10325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:10.146965 10330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:10.146965 10333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:10.146993 10331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:10.147455 10325 server_base.cc:1034] running on GCE node
I20260927 15:03:10.147921 10325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:10.148587 10325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:10.149811 10325 hybrid_clock.cc:648] HybridClock initialized: now 1790521390149772 us; error 47 us; skew 500 ppm
I20260927 15:03:10.151572 10325 webserver.cc:460] Webserver started at http://127.0.98.2:40211/ using document root <none> and password file <none>
I20260927 15:03:10.152155 10325 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:10.152244 10325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:10.152454 10325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:10.154583 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "4e47256c10044689a26e0055b2507a05"
format_stamp: "Formatted at 2026-09-27 15:03:10 on dist-test-slave-ml7t"
server_key: "3dd309b225798becaca21341a9f19ca8"
server_key_iv: "acd04c46fa69b47aff4d00a8fba30a65"
server_key_version: "encryptionkey@0"
I20260927 15:03:10.155138 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4e47256c10044689a26e0055b2507a05"
format_stamp: "Formatted at 2026-09-27 15:03:10 on dist-test-slave-ml7t"
server_key: "3dd309b225798becaca21341a9f19ca8"
server_key_iv: "acd04c46fa69b47aff4d00a8fba30a65"
server_key_version: "encryptionkey@0"
I20260927 15:03:10.158648 10325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:03:10.160881 10339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.162112 10325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:03:10.162225 10325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "4e47256c10044689a26e0055b2507a05"
format_stamp: "Formatted at 2026-09-27 15:03:10 on dist-test-slave-ml7t"
server_key: "3dd309b225798becaca21341a9f19ca8"
server_key_iv: "acd04c46fa69b47aff4d00a8fba30a65"
server_key_version: "encryptionkey@0"
I20260927 15:03:10.162319 10325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:10.166694 10325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:10.167245 10325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:10.167419 10325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:10.167930 10325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:10.168756 10325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:10.168802 10325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.168864 10325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:10.168902 10325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.177838 10325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33677
I20260927 15:03:10.177886 10452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33677 every 8 connection(s)
I20260927 15:03:10.178691 10325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:03:10.182317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10325
I20260927 15:03:10.182446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:03:10.182710 392 external_mini_cluster.cc:1428] Setting key 17f923980f53a1c68688396b83dbb682
I20260927 15:03:10.184574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42229
--builtin_ntp_servers=127.0.98.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:03:10.186834 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:10.180539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41031 (local address 127.0.98.62:42229)
0927 15:03:10.180678 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:10.180682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:10.181673 (+ 991us) server_negotiation.cc:407] Connection header received
0927 15:03:10.183152 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:10.183155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:10.183196 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:10.183313 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:10.186223 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:10.186229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:10.186234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:10.186252 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:10.186354 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:10.186370 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:10.186486 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:10.186610 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:03:10.186663 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:03:10.187680 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:10.180898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42229 (local address 127.0.98.2:41031)
0927 15:03:10.181524 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:10.181563 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:10.182952 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:10.183465 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:10.183472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:10.183558 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:10.185944 (+ 2386us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:03:10.185948 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:10.186007 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:03:10.186076 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:10.186472 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:10.186479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:10.186483 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:03:10.186621 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 15:03:10.186811 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260927 15:03:10.188829 10453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42229
I20260927 15:03:10.189101 10453 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:10.189640 10453 heartbeater.cc:511] Master 127.0.98.62:42229 requested a full tablet report, sending...
I20260927 15:03:10.190755 10134 ts_manager.cc:194] Registered new tserver with Master: 4e47256c10044689a26e0055b2507a05 (127.0.98.2:33677)
W20260927 15:03:10.302656 10456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:10.302968 10456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:10.303104 10456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:10.307602 10456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:10.307695 10456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:10.307802 10456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:03:10.312614 10456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:10.313668 10456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:10.315058 10456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:10.324666 10464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:10.324637 10462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:10.324630 10461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:10.325116 10456 server_base.cc:1034] running on GCE node
I20260927 15:03:10.325508 10456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:10.326153 10456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:10.327306 10456 hybrid_clock.cc:648] HybridClock initialized: now 1790521390327293 us; error 28 us; skew 500 ppm
I20260927 15:03:10.329128 10456 webserver.cc:460] Webserver started at http://127.0.98.3:40841/ using document root <none> and password file <none>
I20260927 15:03:10.329650 10456 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:10.329708 10456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:10.329933 10456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:10.331910 10456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "81f87c93f9c140cdb76bc08acf56488b"
format_stamp: "Formatted at 2026-09-27 15:03:10 on dist-test-slave-ml7t"
server_key: "af0028437947bc725cb00ce921d7ce2a"
server_key_iv: "3eeda7ec063bf2f9840197c5154d5a27"
server_key_version: "encryptionkey@0"
I20260927 15:03:10.332394 10456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81f87c93f9c140cdb76bc08acf56488b"
format_stamp: "Formatted at 2026-09-27 15:03:10 on dist-test-slave-ml7t"
server_key: "af0028437947bc725cb00ce921d7ce2a"
server_key_iv: "3eeda7ec063bf2f9840197c5154d5a27"
server_key_version: "encryptionkey@0"
I20260927 15:03:10.335996 10456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:03:10.338413 10470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.339659 10456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:03:10.339749 10456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "81f87c93f9c140cdb76bc08acf56488b"
format_stamp: "Formatted at 2026-09-27 15:03:10 on dist-test-slave-ml7t"
server_key: "af0028437947bc725cb00ce921d7ce2a"
server_key_iv: "3eeda7ec063bf2f9840197c5154d5a27"
server_key_version: "encryptionkey@0"
I20260927 15:03:10.339838 10456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:10.344524 10456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:10.345084 10456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:10.345217 10456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:10.345702 10456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:10.346609 10456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:10.346655 10456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.346692 10456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:10.346707 10456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:10.356571 10456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46397
I20260927 15:03:10.356590 10583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46397 every 8 connection(s)
I20260927 15:03:10.357509 10456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:03:10.361049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10456
I20260927 15:03:10.361173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:03:10.361449 392 external_mini_cluster.cc:1428] Setting key 852a0269536d9658769a26c30bfde400
I20260927 15:03:10.363780 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:10.359460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60867 (local address 127.0.98.62:42229)
0927 15:03:10.359607 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:10.359610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:10.360717 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 15:03:10.362151 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:10.362154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:10.362195 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:10.362273 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:10.363360 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:10.363363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:10.363366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:03:10.363377 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:10.363400 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:03:10.363410 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:10.363529 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:10.363653 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:03:10.363685 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:03:10.364624 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:10.359855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42229 (local address 127.0.98.3:60867)
0927 15:03:10.360566 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:10.360610 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:10.361951 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:10.362432 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:10.362438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:10.362521 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:10.363046 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:03:10.363048 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:10.363099 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:03:10.363182 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:10.363530 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:10.363538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:10.363544 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:03:10.363686 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:03:10.363896 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260927 15:03:10.365669 10584 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42229
I20260927 15:03:10.365932 10584 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:10.366364 10584 heartbeater.cc:511] Master 127.0.98.62:42229 requested a full tablet report, sending...
I20260927 15:03:10.367202 10134 ts_manager.cc:194] Registered new tserver with Master: 81f87c93f9c140cdb76bc08acf56488b (127.0.98.3:46397)
I20260927 15:03:10.375200 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:03:10.378079 10173 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:03:10.377151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.98.62:42229)
0927 15:03:10.377266 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:10.377270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:10.377469 (+ 199us) server_negotiation.cc:407] Connection header received
0927 15:03:10.377509 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:10.377511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:10.377542 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:10.377615 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:10.377916 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51820: BlockingRecv error: recv got EOF from 127.0.0.1:51820 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:03:10.378147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10194
I20260927 15:03:10.383999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10325
I20260927 15:03:10.389331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10456
I20260927 15:03:10.394586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10103
2026-09-27T15:03:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (828 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:03:10 dist-test-slave-ml7t krb5kdc[10594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:03:10 dist-test-slave-ml7t krb5kdc[10594](info): set up 2 sockets
Sep 27 15:03:10 dist-test-slave-ml7t krb5kdc[10594](info): commencing operation
krb5kdc: starting...
W20260927 15:03:11.437500 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:03:11 dist-test-slave-ml7t krb5kdc[10594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:03:11.570091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:03:11.678136 10607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:11.678406 10607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:11.678447 10607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:11.682446 10607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:03:11.682502 10607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:11.682520 10607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:11.682538 10607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:03:11.682727 10607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:03:11.688896 10607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:11.690030 10607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:11.691072 10607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:11.699574 10613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:11.699599 10612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:11.699684 10615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:11.699879 10607 server_base.cc:1034] running on GCE node
I20260927 15:03:11.700454 10607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:11.701432 10607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:11.702591 10607 hybrid_clock.cc:648] HybridClock initialized: now 1790521391702553 us; error 49 us; skew 500 ppm
Sep 27 15:03:11 dist-test-slave-ml7t krb5kdc[10594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:11.705322 10607 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:03:11.706243 10607 webserver.cc:460] Webserver started at http://127.0.98.62:39361/ using document root <none> and password file <none>
I20260927 15:03:11.706785 10607 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:11.706874 10607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:11.707105 10607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:11.709035 10607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "fc314f8a0db84c1db8e1f6183c98117f"
format_stamp: "Formatted at 2026-09-27 15:03:11 on dist-test-slave-ml7t"
server_key: "11e2af223f8d90d12eed5434c922ab32"
server_key_iv: "4a463d05b983cc72a0a989c3d8ac6cd1"
server_key_version: "encryptionkey@0"
I20260927 15:03:11.709583 10607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc314f8a0db84c1db8e1f6183c98117f"
format_stamp: "Formatted at 2026-09-27 15:03:11 on dist-test-slave-ml7t"
server_key: "11e2af223f8d90d12eed5434c922ab32"
server_key_iv: "4a463d05b983cc72a0a989c3d8ac6cd1"
server_key_version: "encryptionkey@0"
I20260927 15:03:11.713109 10607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:03:11.715484 10622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:11.716594 10607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 15:03:11.716725 10607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "fc314f8a0db84c1db8e1f6183c98117f"
format_stamp: "Formatted at 2026-09-27 15:03:11 on dist-test-slave-ml7t"
server_key: "11e2af223f8d90d12eed5434c922ab32"
server_key_iv: "4a463d05b983cc72a0a989c3d8ac6cd1"
server_key_version: "encryptionkey@0"
I20260927 15:03:11.716830 10607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:11.734683 10607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:11.738092 10607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:11.738286 10607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:11.744976 10607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37845
I20260927 15:03:11.745278 10674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37845 every 8 connection(s)
I20260927 15:03:11.746001 10607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:03:11.748589 10675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:11.753871 10675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f: Bootstrap starting.
I20260927 15:03:11.756096 10675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:11.756173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10607
I20260927 15:03:11.756302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:03:11.756558 392 external_mini_cluster.cc:1428] Setting key 3bc8850815a7bafb04c77e1ee3088118
I20260927 15:03:11.756902 10675 log.cc:824] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:11.758793 10675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f: No bootstrap required, opened a new log
I20260927 15:03:11.762068 10675 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc314f8a0db84c1db8e1f6183c98117f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37845 } }
I20260927 15:03:11.762283 10675 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:11.762347 10675 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc314f8a0db84c1db8e1f6183c98117f, State: Initialized, Role: FOLLOWER
I20260927 15:03:11.762766 10675 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc314f8a0db84c1db8e1f6183c98117f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37845 } }
I20260927 15:03:11.762885 10675 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:03:11.762984 10675 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:03:11.763080 10675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:11.764096 10675 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc314f8a0db84c1db8e1f6183c98117f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37845 } }
I20260927 15:03:11.764438 10675 leader_election.cc:302] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc314f8a0db84c1db8e1f6183c98117f; no voters:
I20260927 15:03:11.764796 10675 leader_election.cc:288] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:03:11.764878 10680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:03:11.765121 10680 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [term 1 LEADER]: Becoming Leader. State: Replica: fc314f8a0db84c1db8e1f6183c98117f, State: Running, Role: LEADER
Sep 27 15:03:11 dist-test-slave-ml7t krb5kdc[10594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:03:11.765455 10680 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc314f8a0db84c1db8e1f6183c98117f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37845 } }
I20260927 15:03:11.766166 10675 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:03:11.767485 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc314f8a0db84c1db8e1f6183c98117f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc314f8a0db84c1db8e1f6183c98117f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37845 } } }
I20260927 15:03:11.767608 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [sys.catalog]: This master's current role is: LEADER
I20260927 15:03:11.767719 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc314f8a0db84c1db8e1f6183c98117f. Latest consensus state: current_term: 1 leader_uuid: "fc314f8a0db84c1db8e1f6183c98117f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc314f8a0db84c1db8e1f6183c98117f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37845 } } }
I20260927 15:03:11.767802 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f [sys.catalog]: This master's current role is: LEADER
I20260927 15:03:11.768632 10689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:03:11.771992 10689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:03:11.772255 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:11.757873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.0.98.62:37845)
0927 15:03:11.758238 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:11.758260 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:11.758293 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:03:11.758944 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:11.758961 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:11.759279 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:11.759598 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:11.760355 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:11.761279 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:11.762217 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:11.762874 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:11.763069 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:11.766034 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:11.766041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:11.766057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:11.766089 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:11.768723 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:11.769311 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:11.769317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:11.769323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:11.769504 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:11.769846 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:11.769855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:11.769857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:11.770397 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:11.770608 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:11.770934 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 15:03:11.771272 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
I20260927 15:03:11.778193 10689 catalog_manager.cc:1347] Generated new cluster ID: ca57c8fd9881472799da4a47dbfeee1c
I20260927 15:03:11.778280 10689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:03:11.795502 10689 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:03:11.796582 10689 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:03:11.803663 10689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f: Generated new TSK 0
I20260927 15:03:11.804358 10689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:03:11.837930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37845
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:03:11.950721 10701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:11.950985 10701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:11.951028 10701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:11.955200 10701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:11.955255 10701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:11.955355 10701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:03:11.960531 10701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:11.961634 10701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:11.962735 10701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:11.972630 10706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:11.972587 10707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:11.972786 10701 server_base.cc:1034] running on GCE node
W20260927 15:03:11.972563 10709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:11.973254 10701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:11.974082 10701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:11.975279 10701 hybrid_clock.cc:648] HybridClock initialized: now 1790521391975245 us; error 45 us; skew 500 ppm
Sep 27 15:03:11 dist-test-slave-ml7t krb5kdc[10594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:11.978195 10701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:03:11.979148 10701 webserver.cc:460] Webserver started at http://127.0.98.1:39981/ using document root <none> and password file <none>
I20260927 15:03:11.979739 10701 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:11.979836 10701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:11.980094 10701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:11.982295 10701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "1c7180147c254d40936cb907c43c8486"
format_stamp: "Formatted at 2026-09-27 15:03:11 on dist-test-slave-ml7t"
server_key: "c82f7f8038d435dcc953edbe5dd9238c"
server_key_iv: "856c0d7dda9fcfae0fe91c1f92fe1de5"
server_key_version: "encryptionkey@0"
I20260927 15:03:11.982903 10701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1c7180147c254d40936cb907c43c8486"
format_stamp: "Formatted at 2026-09-27 15:03:11 on dist-test-slave-ml7t"
server_key: "c82f7f8038d435dcc953edbe5dd9238c"
server_key_iv: "856c0d7dda9fcfae0fe91c1f92fe1de5"
server_key_version: "encryptionkey@0"
I20260927 15:03:11.986709 10701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:03:11.989154 10716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:11.990581 10701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:03:11.990731 10701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "1c7180147c254d40936cb907c43c8486"
format_stamp: "Formatted at 2026-09-27 15:03:11 on dist-test-slave-ml7t"
server_key: "c82f7f8038d435dcc953edbe5dd9238c"
server_key_iv: "856c0d7dda9fcfae0fe91c1f92fe1de5"
server_key_version: "encryptionkey@0"
I20260927 15:03:11.990849 10701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:12.019115 10701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:12.022408 10701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:12.022619 10701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:12.023202 10701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:12.024024 10701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:12.024093 10701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.024163 10701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:12.024217 10701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.033404 10701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36939
I20260927 15:03:12.033432 10829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36939 every 8 connection(s)
I20260927 15:03:12.034353 10701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[10594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:03:12.044668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10701
I20260927 15:03:12.044802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:03:12.045077 392 external_mini_cluster.cc:1428] Setting key e20555aa12fe1ff6e379c79477f309a6
I20260927 15:03:12.047451 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.036119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42115 (local address 127.0.98.62:37845)
0927 15:03:12.036255 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.036259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.037271 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 15:03:12.038106 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.038109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.038154 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.038228 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:12.039393 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.039941 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.041052 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.041267 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.041347 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.043677 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:12.043680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:12.043682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:12.043706 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:12.045342 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.045836 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.045841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.045842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.045925 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.046274 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.046277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.046279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.046489 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:12.046586 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.047061 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.047203 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:03:12.048059 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.036498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37845 (local address 127.0.98.1:42115)
0927 15:03:12.037126 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.037167 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.037927 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.038385 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.038392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.038738 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:12.039257 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.039269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.040062 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.040064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.040945 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.040954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.041394 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.041397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.041528 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.042135 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:12.042149 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:12.043529 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:12.045454 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:12.045464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:12.045474 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:12.045710 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:12.046022 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:12.046027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:12.046029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:12.046165 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:12.046594 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:12.046599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:12.046812 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.046960 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.047198 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":108,"threads_started":1}
I20260927 15:03:12.049376 10830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37845
I20260927 15:03:12.049630 10830 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:12.050148 10830 heartbeater.cc:511] Master 127.0.98.62:37845 requested a full tablet report, sending...
I20260927 15:03:12.051663 10639 ts_manager.cc:194] Registered new tserver with Master: 1c7180147c254d40936cb907c43c8486 (127.0.98.1:36939)
I20260927 15:03:12.052772 10639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42115
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:03:12.072597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37845
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:03:12.181034 10835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:12.181327 10835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:12.181404 10835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:12.185384 10835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:12.185468 10835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:12.185562 10835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:03:12.190714 10835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:12.191727 10835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:12.192719 10835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:12.202450 10841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:12.202495 10843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:12.202725 10835 server_base.cc:1034] running on GCE node
W20260927 15:03:12.202450 10840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:12.203263 10835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:12.203905 10835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:12.205109 10835 hybrid_clock.cc:648] HybridClock initialized: now 1790521392205086 us; error 34 us; skew 500 ppm
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[10594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:12.208038 10835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:03:12.208917 10835 webserver.cc:460] Webserver started at http://127.0.98.2:40297/ using document root <none> and password file <none>
I20260927 15:03:12.209484 10835 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:12.209578 10835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:12.209890 10835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:12.211889 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d7e266a2e934673979ea0c887768608"
format_stamp: "Formatted at 2026-09-27 15:03:12 on dist-test-slave-ml7t"
server_key: "057850e08a3f9ac51310f302f116da81"
server_key_iv: "58d0ef5251e5d97293c322d7528fcd53"
server_key_version: "encryptionkey@0"
I20260927 15:03:12.212440 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d7e266a2e934673979ea0c887768608"
format_stamp: "Formatted at 2026-09-27 15:03:12 on dist-test-slave-ml7t"
server_key: "057850e08a3f9ac51310f302f116da81"
server_key_iv: "58d0ef5251e5d97293c322d7528fcd53"
server_key_version: "encryptionkey@0"
I20260927 15:03:12.216073 10835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 15:03:12.218390 10850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.219532 10835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:03:12.219679 10835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "4d7e266a2e934673979ea0c887768608"
format_stamp: "Formatted at 2026-09-27 15:03:12 on dist-test-slave-ml7t"
server_key: "057850e08a3f9ac51310f302f116da81"
server_key_iv: "58d0ef5251e5d97293c322d7528fcd53"
server_key_version: "encryptionkey@0"
I20260927 15:03:12.219807 10835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:12.233275 10835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:12.236402 10835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:12.236605 10835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:12.237195 10835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:12.238085 10835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:12.238157 10835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.238227 10835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:12.238261 10835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.247529 10835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39031
I20260927 15:03:12.247557 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39031 every 8 connection(s)
I20260927 15:03:12.248418 10835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:03:12.248693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10835
I20260927 15:03:12.248819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:03:12.249179 392 external_mini_cluster.cc:1428] Setting key 2f527acaa015b0ef393ad928db3cf0ab
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[10594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:03:12.263235 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.250311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48103 (local address 127.0.98.62:37845)
0927 15:03:12.250451 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.250454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.251821 (+ 1367us) server_negotiation.cc:407] Connection header received
0927 15:03:12.253012 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.253018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.253076 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.253175 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:12.254394 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.255293 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.256403 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.256790 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.256923 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.259418 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:12.259421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:12.259423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:12.259447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:12.261014 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.261666 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.261669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.261671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.261746 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.262118 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.262123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.262125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.262327 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:12.262436 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.262938 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.263068 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:03:12.264071 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.250785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37845 (local address 127.0.98.2:48103)
0927 15:03:12.251573 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.251635 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.252810 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.253328 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.253334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.253722 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:12.254252 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.254266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.255416 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.255419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.256279 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.256288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.256949 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.256952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.257098 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.257720 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:12.257763 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:12.259259 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:12.261143 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:12.261148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:12.261161 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:12.261534 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:12.261868 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:12.261872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:12.261874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:12.262015 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:12.262433 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:12.262440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:12.262696 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.262896 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.263244 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":105,"threads_started":1}
I20260927 15:03:12.265333 10964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37845
I20260927 15:03:12.265595 10964 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:12.266160 10964 heartbeater.cc:511] Master 127.0.98.62:37845 requested a full tablet report, sending...
I20260927 15:03:12.267354 10639 ts_manager.cc:194] Registered new tserver with Master: 4d7e266a2e934673979ea0c887768608 (127.0.98.2:39031)
I20260927 15:03:12.268033 10639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:03:12.277089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37845
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:03:12.387517 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:12.387771 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:12.387812 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:12.391705 10969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:12.391762 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:12.391819 10969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:03:12.397085 10969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:12.398021 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:12.398944 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:12.408516 10975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:12.408514 10974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:12.408524 10977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:12.408931 10969 server_base.cc:1034] running on GCE node
I20260927 15:03:12.409368 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:12.410027 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:12.411193 10969 hybrid_clock.cc:648] HybridClock initialized: now 1790521392411179 us; error 25 us; skew 500 ppm
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[10594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:12.414062 10969 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:03:12.415007 10969 webserver.cc:460] Webserver started at http://127.0.98.3:43765/ using document root <none> and password file <none>
I20260927 15:03:12.415583 10969 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:12.415679 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:12.415920 10969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:03:12.418231 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e23f57ebf154cf9ac117c0694d627bf"
format_stamp: "Formatted at 2026-09-27 15:03:12 on dist-test-slave-ml7t"
server_key: "0c5daa8acbf0056e62c1d63304ea430a"
server_key_iv: "daa91cb9a4c59e00d1cdb16a96daadf3"
server_key_version: "encryptionkey@0"
I20260927 15:03:12.418943 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e23f57ebf154cf9ac117c0694d627bf"
format_stamp: "Formatted at 2026-09-27 15:03:12 on dist-test-slave-ml7t"
server_key: "0c5daa8acbf0056e62c1d63304ea430a"
server_key_iv: "daa91cb9a4c59e00d1cdb16a96daadf3"
server_key_version: "encryptionkey@0"
I20260927 15:03:12.422924 10969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:03:12.425365 10984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.426779 10969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:03:12.426923 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "1e23f57ebf154cf9ac117c0694d627bf"
format_stamp: "Formatted at 2026-09-27 15:03:12 on dist-test-slave-ml7t"
server_key: "0c5daa8acbf0056e62c1d63304ea430a"
server_key_iv: "daa91cb9a4c59e00d1cdb16a96daadf3"
server_key_version: "encryptionkey@0"
I20260927 15:03:12.427033 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:12.440326 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:12.443200 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:12.443388 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:12.443953 10969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:12.444748 10969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:12.444818 10969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.444888 10969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:12.444933 10969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:12.455554 10969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37583
I20260927 15:03:12.455665 11097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37583 every 8 connection(s)
I20260927 15:03:12.456931 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:03:12.463517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 10969
I20260927 15:03:12.463771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:03:12.464085 392 external_mini_cluster.cc:1428] Setting key 267780a0e1da2f4448ebfc192ec06920
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[10594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:03:12.473084 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.459217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46963 (local address 127.0.98.62:37845)
0927 15:03:12.459359 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.459364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.460872 (+ 1508us) server_negotiation.cc:407] Connection header received
0927 15:03:12.462088 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.462091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.462142 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.462228 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:12.463366 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.464267 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.465924 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.466279 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.466391 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.469088 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:12.469092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:12.469095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:12.469122 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:12.471223 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.471680 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.471682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.471683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.471755 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.472076 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.472079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.472080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.472296 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:12.472391 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.472868 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.472967 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:03:12.473969 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.459753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37845 (local address 127.0.98.3:46963)
0927 15:03:12.460659 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.460720 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.461862 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.462388 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.462394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.462742 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:12.463218 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.463232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.464404 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.464408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.465731 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.465740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.466420 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.466423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.466556 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.467264 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:12.467288 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:12.468888 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:12.471330 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:12.471335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:12.471346 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:12.471587 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:12.471835 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:12.471837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:12.471838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:12.471964 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:12.472402 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:12.472406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:12.472620 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.472795 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.473034 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":146,"threads_started":1}
I20260927 15:03:12.475090 11098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37845
I20260927 15:03:12.475358 11098 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:12.475838 11098 heartbeater.cc:511] Master 127.0.98.62:37845 requested a full tablet report, sending...
I20260927 15:03:12.476902 10639 ts_manager.cc:194] Registered new tserver with Master: 1e23f57ebf154cf9ac117c0694d627bf (127.0.98.3:37583)
I20260927 15:03:12.477563 10639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46963
I20260927 15:03:12.478365 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:03:12.488114 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.480580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.0.98.62:37845)
0927 15:03:12.480735 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.480742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.480940 (+ 198us) server_negotiation.cc:407] Connection header received
0927 15:03:12.481097 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.481100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.481147 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.481236 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:12.481982 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.482481 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.483367 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.483585 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.483700 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.485047 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:12.485052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:12.485060 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:12.485082 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:12.486438 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.486883 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.486891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.486892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.486963 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:12.487242 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:12.487247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:12.487249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:12.487456 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:12.487597 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.487838 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.487948 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:03:12.491143 10638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:03:12.494117 10638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:03:12.507503 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.501996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.0.98.2:39031)
0927 15:03:12.502325 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.502336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.502367 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:03:12.502457 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.502461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.502608 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.502785 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:12.503276 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.504412 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.505953 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.506833 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.506943 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.507086 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.507260 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.507354 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":170,"threads_started":1}
I20260927 15:03:12.507491 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.501869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39031 (local address 127.0.0.1:53448)
0927 15:03:12.502183 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.502227 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.502339 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.502985 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.502993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.503022 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:12.503147 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.503155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.504542 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.504545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.505844 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.505852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.506939 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.506942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.507058 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.507095 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.507219 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.507337 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260927 15:03:12.508208 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.502595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37583 (local address 127.0.0.1:35502)
0927 15:03:12.502797 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.502812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.502889 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.504019 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.504022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.504040 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:12.504144 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.504150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.505519 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.505522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.506734 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.506741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.507800 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.507804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.507919 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.507946 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.508013 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.508078 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":73,"thread_start_us":103,"threads_started":1}
I20260927 15:03:12.508437 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.502902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36939 (local address 127.0.0.1:60686)
0927 15:03:12.503103 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.503115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.503189 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.504155 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.504158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.504187 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:12.504301 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.504309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.505334 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.505340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.506579 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.506589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.508051 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.508054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.508140 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.508162 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.508256 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.508322 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":12,"thread_start_us":83,"threads_started":1}
I20260927 15:03:12.508666 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.502764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.0.98.3:37583)
0927 15:03:12.503540 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.503552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.503574 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:03:12.503650 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.503656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.503851 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.504025 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:12.504257 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.505397 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.506845 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.507664 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.507760 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.507885 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.508382 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.508477 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":93,"threads_started":1}
I20260927 15:03:12.509146 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.503240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.0.98.1:36939)
0927 15:03:12.503608 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.503620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.503637 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:03:12.503697 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.503701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.503850 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.504001 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:12.504430 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.505189 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.506714 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.507549 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.508053 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.508186 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.508911 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.509004 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":111,"threads_started":1}
I20260927 15:03:12.510864 10898 tablet_service.cc:1467] Processing CreateTablet for tablet 896b4ed61d96421d879c5cd84e57fe29 (DEFAULT_TABLE table=test-table [id=c752b584c1c5486b9fbc391ac19de017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:03:12.511245 10764 tablet_service.cc:1467] Processing CreateTablet for tablet 896b4ed61d96421d879c5cd84e57fe29 (DEFAULT_TABLE table=test-table [id=c752b584c1c5486b9fbc391ac19de017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:03:12.511278 11032 tablet_service.cc:1467] Processing CreateTablet for tablet 896b4ed61d96421d879c5cd84e57fe29 (DEFAULT_TABLE table=test-table [id=c752b584c1c5486b9fbc391ac19de017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:03:12.511979 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896b4ed61d96421d879c5cd84e57fe29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:12.512094 10764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896b4ed61d96421d879c5cd84e57fe29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:12.512208 11032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896b4ed61d96421d879c5cd84e57fe29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:12.518958 11115 tablet_bootstrap.cc:492] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: Bootstrap starting.
I20260927 15:03:12.519682 11117 tablet_bootstrap.cc:492] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Bootstrap starting.
I20260927 15:03:12.521200 11115 tablet_bootstrap.cc:654] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:12.521564 11117 tablet_bootstrap.cc:654] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:12.522208 11116 tablet_bootstrap.cc:492] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Bootstrap starting.
I20260927 15:03:12.522389 11115 log.cc:824] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:12.522861 11117 log.cc:824] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:12.524755 11116 tablet_bootstrap.cc:654] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:12.524921 11117 tablet_bootstrap.cc:492] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: No bootstrap required, opened a new log
I20260927 15:03:12.525010 11115 tablet_bootstrap.cc:492] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: No bootstrap required, opened a new log
I20260927 15:03:12.525161 11117 ts_tablet_manager.cc:1397] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:03:12.525203 11115 ts_tablet_manager.cc:1397] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:03:12.525615 11116 log.cc:824] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:12.527524 11116 tablet_bootstrap.cc:492] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: No bootstrap required, opened a new log
I20260927 15:03:12.527704 11116 ts_tablet_manager.cc:1397] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:03:12.528362 11117 raft_consensus.cc:348] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.528651 11117 raft_consensus.cc:374] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:12.528703 11117 raft_consensus.cc:729] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e23f57ebf154cf9ac117c0694d627bf, State: Initialized, Role: FOLLOWER
I20260927 15:03:12.529140 11117 consensus_queue.cc:260] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.529184 11115 raft_consensus.cc:348] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.529452 11115 raft_consensus.cc:374] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:12.529526 11115 raft_consensus.cc:729] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7180147c254d40936cb907c43c8486, State: Initialized, Role: FOLLOWER
I20260927 15:03:12.529978 11115 consensus_queue.cc:260] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.530277 11117 ts_tablet_manager.cc:1428] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 15:03:12.530673 11098 heartbeater.cc:503] Master 127.0.98.62:37845 was elected leader, sending a full tablet report...
I20260927 15:03:12.530922 11115 ts_tablet_manager.cc:1428] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:03:12.531499 11116 raft_consensus.cc:348] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.531760 11116 raft_consensus.cc:374] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:12.531818 11116 raft_consensus.cc:729] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7e266a2e934673979ea0c887768608, State: Initialized, Role: FOLLOWER
I20260927 15:03:12.531847 10830 heartbeater.cc:503] Master 127.0.98.62:37845 was elected leader, sending a full tablet report...
I20260927 15:03:12.532250 11116 consensus_queue.cc:260] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.532909 10964 heartbeater.cc:503] Master 127.0.98.62:37845 was elected leader, sending a full tablet report...
I20260927 15:03:12.533390 11116 ts_tablet_manager.cc:1428] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 15:03:12.535503 10831 tablet.cc:2369] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:03:12.577302 11122 raft_consensus.cc:482] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:03:12.577513 11122 raft_consensus.cc:504] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.578879 11122 leader_election.cc:288] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d7e266a2e934673979ea0c887768608 (127.0.98.2:39031), 1e23f57ebf154cf9ac117c0694d627bf (127.0.98.3:37583)
I20260927 15:03:12.584024 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.579287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46067 (local address 127.0.98.2:39031)
0927 15:03:12.579415 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.579419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.579537 (+ 118us) server_negotiation.cc:407] Connection header received
0927 15:03:12.579723 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.579727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.579785 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.579888 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:12.580264 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.581072 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.582841 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.583389 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.583508 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.583577 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.583813 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.583904 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:03:12.584034 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.579146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39031 (local address 127.0.98.1:46067)
0927 15:03:12.579404 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.579422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.579536 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.579911 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.579914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.579936 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:12.580088 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.580095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.581176 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.581178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.582701 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.582713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.583511 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.583516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.583625 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.583648 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.583790 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.583875 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":84,"thread_start_us":107,"threads_started":1}
I20260927 15:03:12.584100 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.579690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60707 (local address 127.0.98.3:37583)
0927 15:03:12.579804 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.579809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.580032 (+ 223us) server_negotiation.cc:407] Connection header received
0927 15:03:12.580122 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.580127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.580176 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.580277 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:12.580524 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.581217 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.582886 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.583395 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.583472 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.583548 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.583906 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.583975 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:03:12.584460 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.579537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37583 (local address 127.0.98.1:60707)
0927 15:03:12.579925 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.579937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:12.580024 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:12.580305 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:12.580307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:12.580323 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:12.580412 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.580417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.581316 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.581319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.582749 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:12.582760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.583646 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.583650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:12.583787 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.583811 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:03:12.583887 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:03:12.583955 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":107,"thread_start_us":109,"threads_started":1}
I20260927 15:03:12.584846 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896b4ed61d96421d879c5cd84e57fe29" candidate_uuid: "1c7180147c254d40936cb907c43c8486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7e266a2e934673979ea0c887768608" is_pre_election: true
I20260927 15:03:12.585203 10918 raft_consensus.cc:2454] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7180147c254d40936cb907c43c8486 in term 0.
I20260927 15:03:12.585237 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896b4ed61d96421d879c5cd84e57fe29" candidate_uuid: "1c7180147c254d40936cb907c43c8486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e23f57ebf154cf9ac117c0694d627bf" is_pre_election: true
I20260927 15:03:12.585624 11052 raft_consensus.cc:2454] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7180147c254d40936cb907c43c8486 in term 0.
I20260927 15:03:12.585692 10718 leader_election.cc:302] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7180147c254d40936cb907c43c8486, 4d7e266a2e934673979ea0c887768608; no voters:
I20260927 15:03:12.585983 11122 raft_consensus.cc:2781] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:03:12.586073 11122 raft_consensus.cc:482] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:03:12.586129 11122 raft_consensus.cc:3037] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:12.587199 11122 raft_consensus.cc:504] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.587546 11122 leader_election.cc:288] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [CANDIDATE]: Term 1 election: Requested vote from peers 4d7e266a2e934673979ea0c887768608 (127.0.98.2:39031), 1e23f57ebf154cf9ac117c0694d627bf (127.0.98.3:37583)
I20260927 15:03:12.587872 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896b4ed61d96421d879c5cd84e57fe29" candidate_uuid: "1c7180147c254d40936cb907c43c8486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7e266a2e934673979ea0c887768608"
I20260927 15:03:12.587956 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896b4ed61d96421d879c5cd84e57fe29" candidate_uuid: "1c7180147c254d40936cb907c43c8486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e23f57ebf154cf9ac117c0694d627bf"
I20260927 15:03:12.588029 10918 raft_consensus.cc:3037] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:12.588104 11052 raft_consensus.cc:3037] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:12.589092 10918 raft_consensus.cc:2454] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7180147c254d40936cb907c43c8486 in term 1.
I20260927 15:03:12.589118 11052 raft_consensus.cc:2454] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7180147c254d40936cb907c43c8486 in term 1.
I20260927 15:03:12.589375 10718 leader_election.cc:302] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7180147c254d40936cb907c43c8486, 1e23f57ebf154cf9ac117c0694d627bf; no voters:
I20260927 15:03:12.589537 11122 raft_consensus.cc:2781] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:03:12.589756 11122 raft_consensus.cc:686] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7180147c254d40936cb907c43c8486, State: Running, Role: LEADER
I20260927 15:03:12.590134 11122 consensus_queue.cc:237] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } }
I20260927 15:03:12.593266 10639 catalog_manager.cc:5495] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c7180147c254d40936cb907c43c8486 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1c7180147c254d40936cb907c43c8486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7180147c254d40936cb907c43c8486" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36939 } health_report { overall_health: HEALTHY } } }
I20260927 15:03:12.610823 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.607467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.0.98.1:36939)
0927 15:03:12.607600 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:12.607603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:12.607615 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:03:12.607692 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:12.607694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:12.607747 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:12.607836 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:03:12.608069 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.608553 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.609457 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:12.609714 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:12.609854 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:12.610096 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:03:12.610504 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:03:12.610641 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:12.610666 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 15:03:12.610713 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:03:12.616621 11052 raft_consensus.cc:1261] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 1 FOLLOWER]: Refusing update from remote peer 1c7180147c254d40936cb907c43c8486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:03:12.616621 10918 raft_consensus.cc:1261] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 1 FOLLOWER]: Refusing update from remote peer 1c7180147c254d40936cb907c43c8486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:03:12.617339 11122 consensus_queue.cc:1035] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7e266a2e934673979ea0c887768608" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:03:12.617533 11126 consensus_queue.cc:1035] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e23f57ebf154cf9ac117c0694d627bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:03:12.625625 11130 mvcc.cc:205] Tried to move back new op lower bound from 7333975624150339584 to 7333975624051552256. Current Snapshot: MvccSnapshot[applied={T|T < 7333975624150339584}]
I20260927 15:03:12.627483 11131 mvcc.cc:205] Tried to move back new op lower bound from 7333975624150339584 to 7333975624051552256. Current Snapshot: MvccSnapshot[applied={T|T < 7333975624150339584}]
I20260927 15:03:12.633560 10638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:03:12.633736 10638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:03:12.635857 10638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc314f8a0db84c1db8e1f6183c98117f: Sending DeleteTablet for 3 replicas of tablet 896b4ed61d96421d879c5cd84e57fe29
I20260927 15:03:12.636711 10898 tablet_service.cc:1514] Processing DeleteTablet for tablet 896b4ed61d96421d879c5cd84e57fe29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53448
I20260927 15:03:12.636777 11032 tablet_service.cc:1514] Processing DeleteTablet for tablet 896b4ed61d96421d879c5cd84e57fe29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35502
I20260927 15:03:12.637207 11139 tablet_replica.cc:330] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: stopping tablet replica
I20260927 15:03:12.637200 10764 tablet_service.cc:1514] Processing DeleteTablet for tablet 896b4ed61d96421d879c5cd84e57fe29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60686
I20260927 15:03:12.637553 11141 tablet_replica.cc:330] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486: stopping tablet replica
I20260927 15:03:12.637586 11139 raft_consensus.cc:2229] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:03:12.637853 11141 raft_consensus.cc:2229] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:03:12.638022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10701
I20260927 15:03:12.638057 11139 raft_consensus.cc:2258] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:03:12.638229 11141 raft_consensus.cc:2258] T 896b4ed61d96421d879c5cd84e57fe29 P 1c7180147c254d40936cb907c43c8486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:03:12.638504 11140 tablet_replica.cc:330] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: stopping tablet replica
I20260927 15:03:12.638852 11140 raft_consensus.cc:2229] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:03:12.639163 11140 raft_consensus.cc:2258] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:03:12.640692 11140 ts_tablet_manager.cc:1905] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:03:12.642505 11139 ts_tablet_manager.cc:1905] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:03:12.644124 11140 ts_tablet_manager.cc:1918] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:03:12.644233 11140 log.cc:1196] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/896b4ed61d96421d879c5cd84e57fe29
I20260927 15:03:12.644589 11140 ts_tablet_manager.cc:1939] T 896b4ed61d96421d879c5cd84e57fe29 P 1e23f57ebf154cf9ac117c0694d627bf: Deleting consensus metadata
I20260927 15:03:12.644622 11139 ts_tablet_manager.cc:1918] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:03:12.644730 11139 log.cc:1196] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/896b4ed61d96421d879c5cd84e57fe29
I20260927 15:03:12.644994 11139 ts_tablet_manager.cc:1939] T 896b4ed61d96421d879c5cd84e57fe29 P 4d7e266a2e934673979ea0c887768608: Deleting consensus metadata
I20260927 15:03:12.645872 10624 catalog_manager.cc:4841] TS 1e23f57ebf154cf9ac117c0694d627bf (127.0.98.3:37583): tablet 896b4ed61d96421d879c5cd84e57fe29 (table test-table [id=c752b584c1c5486b9fbc391ac19de017]) successfully deleted
W20260927 15:03:12.646157 10624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36939 (error 108)
I20260927 15:03:12.646322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10835
I20260927 15:03:12.646689 10624 catalog_manager.cc:4841] TS 4d7e266a2e934673979ea0c887768608 (127.0.98.2:39031): tablet 896b4ed61d96421d879c5cd84e57fe29 (table test-table [id=c752b584c1c5486b9fbc391ac19de017]) successfully deleted
I20260927 15:03:12.647627 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:12.647323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36939 (local address 127.0.0.1:60698)
0927 15:03:12.647455 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:12.647528 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:03:12.647974 10624 catalog_manager.cc:4584] TS 1c7180147c254d40936cb907c43c8486 (127.0.98.1:36939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 896b4ed61d96421d879c5cd84e57fe29: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36939: connect: Connection refused (error 111)
I20260927 15:03:12.652019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10969
I20260927 15:03:12.657263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 10607
2026-09-27T15:03:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2263 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[11145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[11145](info): set up 2 sockets
Sep 27 15:03:12 dist-test-slave-ml7t krb5kdc[11145](info): commencing operation
krb5kdc: starting...
W20260927 15:03:13.698850 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:03:13 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:03:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 15:03:13.864320 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYaV7G8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/postgres -l logfile start
2026-09-27 15:03:16.385 UTC [11175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:03:16.385 UTC [11175] LOG: listening on IPv4 address "127.0.98.20", port 49327
2026-09-27 15:03:16.388 UTC [11175] LOG: listening on Unix socket "/tmp/.s.PGSQL.49327"
2026-09-27 15:03:16.412 UTC [11178] LOG: database system was shut down at 2026-09-27 15:03:14 UTC
2026-09-27 15:03:16.415 UTC [11175] LOG: database system is ready to accept connections
I20260927 15:03:17.382961 392 mini_postgres.cc:96] Postgres bound to 49327
2026-09-27 15:03:17.389 UTC [11186] FATAL: database "slave" does not exist
127.0.98.20:49327 - accepting connections
I20260927 15:03:17.390218 392 mini_ranger.cc:161] Starting Ranger...
I20260927 15:03:17.404804 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:03:17.878679 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:03:17.878777 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-admin
2026-09-27 15:03:18,164 [I] DB FLAVOR :POSTGRES
2026-09-27 15:03:18,164 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:03:18,164 [I] Checking connection..
2026-09-27 15:03:18,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:03:18,418 [I] Checking connection passed.
2026-09-27 15:03:18,419 [I] --------- Verifying version history table ---------
2026-09-27 15:03:18,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:03:18,651 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:03:18,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:03:18,887 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:03:18,888 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:03:18,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:03:19.119 UTC [11250] WARNING: there is no transaction in progress
2026-09-27 15:03:19,125 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:03:19,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:03:19,397 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:03:19,397 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:03:19,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:03:19,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:03:19,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ml7t.c.gcp-upstream.internal','N') ;"
2026-09-27 15:03:20,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:03:20,399 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:03:20,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:03:20,654 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:03:20,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:03:20,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:03:21,167 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:03:21,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:03:22.162 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.180 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.197 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.214 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.230 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.250 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.305 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.312 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.326 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.336 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.346 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.359 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.369 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.375 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.382 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:22.389 UTC [11358] WARNING: there is no transaction in progress
2026-09-27 15:03:23,027 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:03:23,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:03:23,291 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:03:23,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:03:23,557 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:03:23,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:03:23,801 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:03:23,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:03:24,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:03:24,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ml7t.c.gcp-upstream.internal';"
2026-09-27 15:03:24,536 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:03:24,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:03:24,811 [I] DB_PATCHES have already been applied
I20260927 15:03:24.817039 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYaV7G8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:03:24.817103 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 15:03:24.820447 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39179
Sep 27, 2026 3:03:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:03:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:03:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:03:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:03:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:03:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:03:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:03:25.938495 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.548s user 0.001s sys 0.009s
Sep 27, 2026 3:03:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:03:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:03:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:03:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:03:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:03:47.850970 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:03:47 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521427, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:03:47.940151 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:03:47.950409 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:03:48.816165 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:03:48.816342 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms
I20260927 15:03:48.816457 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49327
2026-09-27 15:03:48,996 [I] DB FLAVOR :POSTGRES
2026-09-27 15:03:48,996 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:03:48,996 [I] Checking connection
2026-09-27 15:03:48,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:03:49,249 [I] connection success
2026-09-27 15:03:49,250 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:03:49,250 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:03:49,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:03:49,488 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:03:49,489 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:03:49,489 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:03:49,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:03:49,741 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:03:50.233886 392 mini_ranger_kms.cc:310] Created kms service
I20260927 15:03:50.421667 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:03:50.466320 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:03:50 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521427, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:03:50.884640 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:03:50.884794 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:03:50.884820 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 15:03:50.888319 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11538
I20260927 15:03:50.888474 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:40037
Sep 27, 2026 3:03:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:03:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:03:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:03:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:03:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:03:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:03:53.184113 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.244s user 0.009s sys 0.156s
I20260927 15:03:53.184260 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:03:53.184362 392 mini_ranger_kms.cc:401] 127.0.98.20:40037/kms/v1/keys
Sep 27 15:03:53 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521433, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:03:53 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521433, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 15:03:54 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521433, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 15:03:56 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:03:56.461642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44363
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44363
--ranger_config_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40037/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:03:56.574752 11589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:56.575021 11589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:56.575060 11589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:56.579260 11589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:03:56.579324 11589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:56.579344 11589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:56.579360 11589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:03:56.579375 11589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:03:56.584976 11589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:40037/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44363
--ranger_config_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:56.586046 11589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:56.587116 11589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:56.596474 11597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:56.596792 11589 server_base.cc:1034] running on GCE node
W20260927 15:03:56.596585 11594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:56.596483 11595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:56.597702 11589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:56.599174 11589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:56.600379 11589 hybrid_clock.cc:648] HybridClock initialized: now 1790521436600348 us; error 40 us; skew 500 ppm
Sep 27 15:03:56 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:56.603781 11589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:03:56.604691 11589 webserver.cc:460] Webserver started at http://127.0.98.62:41227/ using document root <none> and password file <none>
I20260927 15:03:56.605265 11589 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:56.605345 11589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:56.605556 11589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:03:56 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:03:56.657450 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "bda69f92286f4c46a1a8499fbba6ea7d"
format_stamp: "Formatted at 2026-09-27 15:03:56 on dist-test-slave-ml7t"
server_key: "f4300684afefccba1d83998a47988c6e"
server_key_iv: "c7491101d933aab314d3c8a6f71e080e"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:56.658152 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "bda69f92286f4c46a1a8499fbba6ea7d"
format_stamp: "Formatted at 2026-09-27 15:03:56 on dist-test-slave-ml7t"
server_key: "f4300684afefccba1d83998a47988c6e"
server_key_iv: "c7491101d933aab314d3c8a6f71e080e"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:56.662526 11589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:03:56.679463 11604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:56.681146 11589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:03:56.681286 11589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "bda69f92286f4c46a1a8499fbba6ea7d"
format_stamp: "Formatted at 2026-09-27 15:03:56 on dist-test-slave-ml7t"
server_key: "f4300684afefccba1d83998a47988c6e"
server_key_iv: "c7491101d933aab314d3c8a6f71e080e"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:56.681420 11589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:56.702267 11589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:56.705549 11589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:56.705760 11589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:56.714586 11589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44363
I20260927 15:03:56.714591 11656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44363 every 8 connection(s)
I20260927 15:03:56.715747 11589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:03:56.718602 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:56.719208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 11589
I20260927 15:03:56.719341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:03:56.719641 392 external_mini_cluster.cc:1428] Setting key de1a2cae85c5e69037a9b3a06db2a644
I20260927 15:03:56.725041 11657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d: Bootstrap starting.
I20260927 15:03:56.727494 11657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:56.728408 11657 log.cc:824] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:03:56 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:03:56.730535 11657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d: No bootstrap required, opened a new log
I20260927 15:03:56.733816 11657 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260927 15:03:56.734092 11657 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:56.734177 11657 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda69f92286f4c46a1a8499fbba6ea7d, State: Initialized, Role: FOLLOWER
I20260927 15:03:56.734642 11657 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260927 15:03:56.734797 11657 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:03:56.734921 11657 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:03:56.735070 11657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:56.736099 11657 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260927 15:03:56.736439 11657 leader_election.cc:302] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda69f92286f4c46a1a8499fbba6ea7d; no voters:
I20260927 15:03:56.736562 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:56.721080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.0.98.62:44363)
0927 15:03:56.721549 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:56.721572 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:56.721629 (+ 57us) server_negotiation.cc:407] Connection header received
0927 15:03:56.722482 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:56.722505 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:56.722943 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:56.723351 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:56.724163 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:56.725569 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:56.726814 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:56.727504 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:56.727658 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:56.730761 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:56.730767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:56.730802 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:56.730830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:56.733414 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:56.733939 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:56.733945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:56.733951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:56.734077 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:56.734415 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:56.734419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:56.734420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:56.734892 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:56.735090 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:56.735390 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 15:03:56.735653 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
I20260927 15:03:56.736776 11657 leader_election.cc:288] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:03:56.737018 11662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:03:56.737262 11662 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [term 1 LEADER]: Becoming Leader. State: Replica: bda69f92286f4c46a1a8499fbba6ea7d, State: Running, Role: LEADER
I20260927 15:03:56.737596 11662 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260927 15:03:56.739002 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } } }
I20260927 15:03:56.739117 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [sys.catalog]: This master's current role is: LEADER
I20260927 15:03:56.739120 11657 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:03:56.739883 11666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:03:56.741145 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda69f92286f4c46a1a8499fbba6ea7d. Latest consensus state: current_term: 1 leader_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda69f92286f4c46a1a8499fbba6ea7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } } }
I20260927 15:03:56.741340 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d [sys.catalog]: This master's current role is: LEADER
I20260927 15:03:56.743642 11666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:03:56.744190 11657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:03:56.753886 11666 catalog_manager.cc:1347] Generated new cluster ID: a3c9bdc6846b4103b3671dc0c15ec6fb
I20260927 15:03:56.754011 11666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:03:56.762333 11666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:03:56.763684 11666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:03:56.776070 11666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d: Generated new TSK 0
I20260927 15:03:56.777021 11666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:03:57.517588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44363
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:03:57.639554 11702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:57.639992 11702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:57.640122 11702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:57.645804 11702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:57.645893 11702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:57.645998 11702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:03:57.651364 11702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:40037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:57.652479 11702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:57.653510 11702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:57.663067 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:57.663067 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:57.663466 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:57.663822 11702 server_base.cc:1034] running on GCE node
I20260927 15:03:57.664301 11702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:57.664919 11702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:57.666102 11702 hybrid_clock.cc:648] HybridClock initialized: now 1790521437666074 us; error 46 us; skew 500 ppm
Sep 27 15:03:57 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:57.669147 11702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:03:57.670218 11702 webserver.cc:460] Webserver started at http://127.0.98.1:32917/ using document root <none> and password file <none>
I20260927 15:03:57.670811 11702 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:57.670862 11702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:57.671026 11702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:03:57 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:03:57.698251 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "91100cd166fe4806b89e404940eeae8b"
format_stamp: "Formatted at 2026-09-27 15:03:57 on dist-test-slave-ml7t"
server_key: "e0ab9e1b5e52b34f9ccd6e7bdfd9b1cf"
server_key_iv: "f6f69d7261107d8bef47173ecf8d83be"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:57.699117 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91100cd166fe4806b89e404940eeae8b"
format_stamp: "Formatted at 2026-09-27 15:03:57 on dist-test-slave-ml7t"
server_key: "e0ab9e1b5e52b34f9ccd6e7bdfd9b1cf"
server_key_iv: "f6f69d7261107d8bef47173ecf8d83be"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:57.707973 11702 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 15:03:57.730224 11719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:57.732096 11702 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260927 15:03:57.732247 11702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "91100cd166fe4806b89e404940eeae8b"
format_stamp: "Formatted at 2026-09-27 15:03:57 on dist-test-slave-ml7t"
server_key: "e0ab9e1b5e52b34f9ccd6e7bdfd9b1cf"
server_key_iv: "f6f69d7261107d8bef47173ecf8d83be"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:57.732383 11702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:57.758725 11702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:57.768270 11702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:57.768522 11702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:57.769249 11702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:57.770396 11702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:57.770501 11702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:57.770649 11702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:57.770712 11702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:57.784471 11702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35679
I20260927 15:03:57.784518 11833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35679 every 8 connection(s)
I20260927 15:03:57.785394 11702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:03:57.786257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 11702
I20260927 15:03:57.786371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:03:57.786669 392 external_mini_cluster.cc:1428] Setting key ca81b43174789965b6e74451f5f39be5
Sep 27 15:03:57 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:03:57.809341 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:57.792396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44363 (local address 127.0.98.1:55333)
0927 15:03:57.793959 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:57.794015 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:57.795144 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:57.796064 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:57.796073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:57.796432 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:57.797110 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:57.797131 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:57.798314 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:57.798317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:57.799754 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:57.799766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:57.800548 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:57.800558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:57.800774 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:57.801505 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:57.801526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:57.803328 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:57.805669 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:57.805675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:57.805694 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:57.806087 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:57.806454 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:57.806457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:57.806459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:57.806648 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:57.807170 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:57.807177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:57.807523 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 15:03:57.807749 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 15:03:57.808110 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"thread_start_us":160,"threads_started":1}
I20260927 15:03:57.810250 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:57.791829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55333 (local address 127.0.98.62:44363)
0927 15:03:57.792091 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:57.792095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:57.795466 (+ 3371us) server_negotiation.cc:407] Connection header received
0927 15:03:57.795519 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:57.795522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:57.795579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:57.795742 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:57.797290 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:57.798152 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:57.799901 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:57.800254 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:57.800358 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:57.803545 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:57.803549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:57.803552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:57.803581 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:57.805530 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:57.806216 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:57.806219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:57.806221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:57.806325 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:57.806778 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:57.806782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:57.806784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:57.807023 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:57.807117 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:57.809913 (+ 2796us) server_negotiation.cc:299] Negotiation successful
0927 15:03:57.810061 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
I20260927 15:03:57.812177 11835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44363
I20260927 15:03:57.812434 11835 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:03:57.813103 11835 heartbeater.cc:511] Master 127.0.98.62:44363 requested a full tablet report, sending...
I20260927 15:03:57.814971 11621 ts_manager.cc:194] Registered new tserver with Master: 91100cd166fe4806b89e404940eeae8b (127.0.98.1:35679)
I20260927 15:03:57.816383 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:03:57.833340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44363
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:03:57 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:03:57.984799 11841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:57.985236 11841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:57.985319 11841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:03:57.991786 11841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:57.991892 11841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:57.992000 11841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:03:58.000303 11841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:40037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:58.001736 11841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:58.003163 11841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:58.016814 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:58.016872 11850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:58.016851 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:58.017601 11841 server_base.cc:1034] running on GCE node
I20260927 15:03:58.018100 11841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:58.018826 11841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:58.020035 11841 hybrid_clock.cc:648] HybridClock initialized: now 1790521438020002 us; error 48 us; skew 500 ppm
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:58.024286 11841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:03:58.025445 11841 webserver.cc:460] Webserver started at http://127.0.98.2:40573/ using document root <none> and password file <none>
I20260927 15:03:58.026216 11841 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:58.026304 11841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:58.026537 11841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:03:58.047248 11841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "970184cee94442c782e4349f96d15d98"
format_stamp: "Formatted at 2026-09-27 15:03:58 on dist-test-slave-ml7t"
server_key: "f7a6cd20bca45faf20b4345ff0eb586f"
server_key_iv: "c95048435efd5384ffc7e6c3cc3e6a9a"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:58.048152 11841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "970184cee94442c782e4349f96d15d98"
format_stamp: "Formatted at 2026-09-27 15:03:58 on dist-test-slave-ml7t"
server_key: "f7a6cd20bca45faf20b4345ff0eb586f"
server_key_iv: "c95048435efd5384ffc7e6c3cc3e6a9a"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:58.058979 11841 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260927 15:03:58.079712 11857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:58.081633 11841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 15:03:58.081848 11841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "970184cee94442c782e4349f96d15d98"
format_stamp: "Formatted at 2026-09-27 15:03:58 on dist-test-slave-ml7t"
server_key: "f7a6cd20bca45faf20b4345ff0eb586f"
server_key_iv: "c95048435efd5384ffc7e6c3cc3e6a9a"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:58.082016 11841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:58.111388 11841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:58.115046 11841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:58.115236 11841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:58.115881 11841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:58.116880 11841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:58.116943 11841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:58.117000 11841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:58.117029 11841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:58.131187 11841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41771
I20260927 15:03:58.132328 11841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:03:58.133623 11970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41771 every 8 connection(s)
I20260927 15:03:58.142011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 11841
I20260927 15:03:58.142158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:03:58.142895 392 external_mini_cluster.cc:1428] Setting key dd8ce70a968e75850a9e1e75dac17245
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:03:58.150386 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.135297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56311 (local address 127.0.98.62:44363)
0927 15:03:58.135461 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.135466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.138066 (+ 2600us) server_negotiation.cc:407] Connection header received
0927 15:03:58.139192 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.139196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.139252 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.139339 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:58.140655 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.141525 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.142953 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.143190 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.143304 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.146648 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:58.146654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:58.146660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:58.146721 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:58.148215 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:58.148746 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:58.148750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:58.148752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:58.148832 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:58.149183 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:58.149186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:58.149188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:58.149366 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:58.149492 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.150064 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.150192 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:03:58.151315 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.135699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44363 (local address 127.0.98.2:56311)
0927 15:03:58.137904 (+ 2205us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.137952 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:58.138974 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:58.139529 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:58.139536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:58.139965 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:58.140509 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.140523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.141741 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.141745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.142827 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.142836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.143748 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.143754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.143945 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.144820 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:58.144837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:58.146434 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:58.148354 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:58.148360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:58.148376 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:58.148639 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:58.148931 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:58.148934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:58.148936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:58.149082 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:58.149490 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:58.149497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:58.149808 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 15:03:58.150010 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 15:03:58.150333 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1947,"thread_start_us":162,"threads_started":1}
I20260927 15:03:58.152700 11971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44363
I20260927 15:03:58.152956 11971 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:58.153462 11971 heartbeater.cc:511] Master 127.0.98.62:44363 requested a full tablet report, sending...
I20260927 15:03:58.154626 11621 ts_manager.cc:194] Registered new tserver with Master: 970184cee94442c782e4349f96d15d98 (127.0.98.2:41771)
I20260927 15:03:58.155254 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56311
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:03:58.182504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44363
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:03:58.338111 11976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:03:58.338439 11976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:03:58.338498 11976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 15:03:58.344419 11976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:03:58.344498 11976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:03:58.344583 11976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:03:58.352366 11976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:40037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:03:58.353742 11976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:03:58.355058 11976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:03:58.367707 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:58.368050 11984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:03:58.368542 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:03:58.369550 11976 server_base.cc:1034] running on GCE node
I20260927 15:03:58.370047 11976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:03:58.370884 11976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:03:58.374229 11976 hybrid_clock.cc:648] HybridClock initialized: now 1790521438374152 us; error 81 us; skew 500 ppm
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:03:58.377959 11976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:03:58.379047 11976 webserver.cc:460] Webserver started at http://127.0.98.3:35765/ using document root <none> and password file <none>
I20260927 15:03:58.379746 11976 fs_manager.cc:359] Metadata directory not provided
I20260927 15:03:58.379834 11976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:03:58.380082 11976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:03:58.407573 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "8248e297a6b448f99ec43c55d912ae5f"
format_stamp: "Formatted at 2026-09-27 15:03:58 on dist-test-slave-ml7t"
server_key: "c9465b399664fe0b4ac60a9794443903"
server_key_iv: "ee94b2198b7cdaff3736a455c003a343"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:58.408372 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8248e297a6b448f99ec43c55d912ae5f"
format_stamp: "Formatted at 2026-09-27 15:03:58 on dist-test-slave-ml7t"
server_key: "c9465b399664fe0b4ac60a9794443903"
server_key_iv: "ee94b2198b7cdaff3736a455c003a343"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:58.426918 11976 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.007s sys 0.000s
I20260927 15:03:58.460767 11991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:58.462481 11976 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 15:03:58.462675 11976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "8248e297a6b448f99ec43c55d912ae5f"
format_stamp: "Formatted at 2026-09-27 15:03:58 on dist-test-slave-ml7t"
server_key: "c9465b399664fe0b4ac60a9794443903"
server_key_iv: "ee94b2198b7cdaff3736a455c003a343"
server_key_version: "kuduclusterkey@0"
I20260927 15:03:58.462898 11976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:03:58.482086 11976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:03:58.486748 11976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:03:58.486958 11976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:03:58.487633 11976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:03:58.488703 11976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:03:58.488771 11976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:58.488840 11976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:03:58.488875 11976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:03:58.499802 11976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44745
I20260927 15:03:58.501017 11976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:03:58.505218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 11976
I20260927 15:03:58.505352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:03:58.505640 392 external_mini_cluster.cc:1428] Setting key e36c7113bc4ed42160ec20bdbe6e1329
I20260927 15:03:58.517263 12104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44745 every 8 connection(s)
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:03:58.534617 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.517003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45763 (local address 127.0.98.62:44363)
0927 15:03:58.517120 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.517124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.517885 (+ 761us) server_negotiation.cc:407] Connection header received
0927 15:03:58.520993 (+ 3108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.520997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.521058 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.521111 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:58.522918 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.523755 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.525456 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.525845 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.525930 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.529109 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:58.529113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:58.529116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:58.529143 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:58.530909 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:58.532712 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:58.532715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:58.532717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:58.532806 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:58.533227 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:58.533230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:58.533232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:58.533465 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:58.533539 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.534331 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.534454 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:03:58.536504 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.515873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44363 (local address 127.0.98.3:45763)
0927 15:03:58.517680 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.517746 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:58.520731 (+ 2985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:58.521548 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:58.521558 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:58.522014 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:03:58.522741 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.522764 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.523909 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.523912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.525319 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.525329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.526133 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.526137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.526328 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.527105 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:03:58.527126 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:03:58.528919 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:03:58.531604 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:58.531609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:58.531626 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:58.532579 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:58.532921 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:03:58.532924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:03:58.532926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:03:58.533112 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:03:58.533621 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:03:58.533627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:03:58.534024 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 15:03:58.535073 (+ 1049us) client_negotiation.cc:240] Negotiation successful
0927 15:03:58.535435 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1591,"thread_start_us":168,"threads_started":1}
I20260927 15:03:58.538066 12105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44363
I20260927 15:03:58.538297 12105 heartbeater.cc:464] Registering TS with master...
I20260927 15:03:58.538920 12105 heartbeater.cc:511] Master 127.0.98.62:44363 requested a full tablet report, sending...
I20260927 15:03:58.540082 11621 ts_manager.cc:194] Registered new tserver with Master: 8248e297a6b448f99ec43c55d912ae5f (127.0.98.3:44745)
I20260927 15:03:58.540858 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45763
I20260927 15:03:58.544384 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:03:58.556502 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.546600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.0.98.62:44363)
0927 15:03:58.546709 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.546714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.547260 (+ 546us) server_negotiation.cc:407] Connection header received
0927 15:03:58.547451 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.547454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.547498 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.547549 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:03:58.548527 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.549324 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.550639 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.550965 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.551042 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.552821 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:03:58.552825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:03:58.552828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:03:58.552852 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:03:58.554611 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:58.555098 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:58.555101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:58.555103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:58.555184 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:03:58.555523 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:03:58.555526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:03:58.555528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:03:58.555742 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:03:58.555846 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.556213 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.556334 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:03:58.560091 11621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:03:58.564878 11621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:03:58.603448 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.580991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44745 (local address 127.0.0.1:34722)
0927 15:03:58.584155 (+ 3164us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.584167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:58.584265 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:58.585736 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:58.585742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:58.585789 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:58.585962 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.585971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.591557 (+ 5586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.591560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.593786 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.593798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.602131 (+ 8333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.602135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.602992 (+ 857us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.603036 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 15:03:58.603145 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:03:58.603285 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3108,"thread_start_us":107,"threads_started":1}
I20260927 15:03:58.604204 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.583075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41771 (local address 127.0.0.1:34048)
0927 15:03:58.586710 (+ 3635us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.586723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:58.586811 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:58.588883 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:58.588886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:58.588905 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:58.589009 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.589015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.594623 (+ 5608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.594627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.595882 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.595893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.603811 (+ 7918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.603815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.603918 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.603942 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:03:58.604013 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:03:58.604079 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3291,"thread_start_us":77,"threads_started":1}
I20260927 15:03:58.605794 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.578444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35679 (local address 127.0.0.1:43468)
0927 15:03:58.583591 (+ 5147us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.583640 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:58.583761 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:58.590318 (+ 6557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:58.590321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:58.590341 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:58.590427 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.590434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.598164 (+ 7730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.598167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.599469 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.599482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.605406 (+ 5924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.605410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.605499 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.605520 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:03:58.605587 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:03:58.605649 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4977,"thread_start_us":115,"threads_started":1}
I20260927 15:03:58.607029 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.582080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.0.98.2:41771)
0927 15:03:58.584480 (+ 2400us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.584495 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.586927 (+ 2432us) server_negotiation.cc:407] Connection header received
0927 15:03:58.587013 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.587069 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.587323 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.587564 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:58.589139 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.594489 (+ 5350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.599675 (+ 5186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.600558 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.600667 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.600832 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.606780 (+ 5948us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.606892 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2303,"thread_start_us":135,"threads_started":1}
I20260927 15:03:58.607527 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.581433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.0.98.3:44745)
0927 15:03:58.584031 (+ 2598us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.584042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.584536 (+ 494us) server_negotiation.cc:407] Connection header received
0927 15:03:58.584619 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.584624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.584839 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.584989 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:58.586207 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.591424 (+ 5217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.600926 (+ 9502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.601822 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.601931 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.602068 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.607275 (+ 5207us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.607387 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":129,"threads_started":1}
I20260927 15:03:58.610925 11905 tablet_service.cc:1467] Processing CreateTablet for tablet 82b9ac2fb0af4779aaa561152b883e70 (DEFAULT_TABLE table=test-table [id=70c2e2b652804aba9d1c8962bbda2bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:03:58.611315 12039 tablet_service.cc:1467] Processing CreateTablet for tablet 82b9ac2fb0af4779aaa561152b883e70 (DEFAULT_TABLE table=test-table [id=70c2e2b652804aba9d1c8962bbda2bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:03:58.612076 11905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b9ac2fb0af4779aaa561152b883e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:58.612179 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b9ac2fb0af4779aaa561152b883e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:58.621850 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.579401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.0.98.1:35679)
0927 15:03:58.586587 (+ 7186us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.586593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.588008 (+ 1415us) server_negotiation.cc:407] Connection header received
0927 15:03:58.588098 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.588103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.590019 (+ 1916us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.590162 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:58.596036 (+ 5874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.598033 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.604300 (+ 6267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.605104 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.605208 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.605346 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.621532 (+ 16186us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.621671 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7085,"thread_start_us":66,"threads_started":1}
I20260927 15:03:58.625135 11764 tablet_service.cc:1467] Processing CreateTablet for tablet 82b9ac2fb0af4779aaa561152b883e70 (DEFAULT_TABLE table=test-table [id=70c2e2b652804aba9d1c8962bbda2bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:03:58.627312 11764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b9ac2fb0af4779aaa561152b883e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:03:58.637328 12124 tablet_bootstrap.cc:492] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: Bootstrap starting.
I20260927 15:03:58.637782 12126 tablet_bootstrap.cc:492] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b: Bootstrap starting.
I20260927 15:03:58.642381 12124 tablet_bootstrap.cc:654] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:58.643059 12126 tablet_bootstrap.cc:654] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:58.643930 12126 log.cc:824] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:58.644973 12125 tablet_bootstrap.cc:492] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98: Bootstrap starting.
I20260927 15:03:58.646291 12124 log.cc:824] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:58.648268 12125 tablet_bootstrap.cc:654] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98: Neither blocks nor log segments found. Creating new log.
I20260927 15:03:58.650604 12125 log.cc:824] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98: Log is configured to *not* fsync() on all Append() calls
I20260927 15:03:58.652998 12126 tablet_bootstrap.cc:492] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b: No bootstrap required, opened a new log
I20260927 15:03:58.653241 12126 ts_tablet_manager.cc:1397] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260927 15:03:58.657995 12126 raft_consensus.cc:348] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.660385 12124 tablet_bootstrap.cc:492] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: No bootstrap required, opened a new log
I20260927 15:03:58.660631 12124 ts_tablet_manager.cc:1397] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20260927 15:03:58.665992 12124 raft_consensus.cc:348] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.666357 12124 raft_consensus.cc:374] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:58.666453 12124 raft_consensus.cc:729] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8248e297a6b448f99ec43c55d912ae5f, State: Initialized, Role: FOLLOWER
I20260927 15:03:58.666815 12126 raft_consensus.cc:374] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:58.666952 12126 raft_consensus.cc:729] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91100cd166fe4806b89e404940eeae8b, State: Initialized, Role: FOLLOWER
I20260927 15:03:58.667323 12124 consensus_queue.cc:260] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.668118 12126 consensus_queue.cc:260] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.670317 12126 ts_tablet_manager.cc:1428] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b: Time spent starting tablet: real 0.017s user 0.002s sys 0.004s
I20260927 15:03:58.669732 12124 ts_tablet_manager.cc:1428] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 15:03:58.671298 12105 heartbeater.cc:503] Master 127.0.98.62:44363 was elected leader, sending a full tablet report...
I20260927 15:03:58.671779 11835 heartbeater.cc:503] Master 127.0.98.62:44363 was elected leader, sending a full tablet report...
I20260927 15:03:58.673529 12125 tablet_bootstrap.cc:492] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98: No bootstrap required, opened a new log
I20260927 15:03:58.673910 12125 ts_tablet_manager.cc:1397] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.002s
I20260927 15:03:58.679620 12125 raft_consensus.cc:348] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.680035 12125 raft_consensus.cc:374] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:03:58.680152 12125 raft_consensus.cc:729] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970184cee94442c782e4349f96d15d98, State: Initialized, Role: FOLLOWER
I20260927 15:03:58.680819 12125 consensus_queue.cc:260] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.681953 12125 ts_tablet_manager.cc:1428] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260927 15:03:58.682367 11971 heartbeater.cc:503] Master 127.0.98.62:44363 was elected leader, sending a full tablet report...
I20260927 15:03:58.686582 12130 raft_consensus.cc:482] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:03:58.686780 12130 raft_consensus.cc:504] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.694653 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.690279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35679 (local address 127.0.98.3:58877)
0927 15:03:58.690391 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.690408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:58.690494 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:58.690952 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:58.690955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:58.690973 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:58.691085 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.691091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.692064 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.692067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.693257 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.693267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.694239 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.694243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.694348 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.694372 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:03:58.694448 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:03:58.694519 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 15:03:58.695294 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.690540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58877 (local address 127.0.98.1:35679)
0927 15:03:58.690652 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.690656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.690669 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:03:58.690712 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.690714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.690755 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.690840 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:58.691213 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.691934 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.693394 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.693981 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.694098 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.694173 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.695105 (+ 932us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.695183 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:03:58.695981 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b9ac2fb0af4779aaa561152b883e70" candidate_uuid: "8248e297a6b448f99ec43c55d912ae5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91100cd166fe4806b89e404940eeae8b" is_pre_election: true
I20260927 15:03:58.696301 11787 raft_consensus.cc:2454] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8248e297a6b448f99ec43c55d912ae5f in term 0.
I20260927 15:03:58.696903 11993 leader_election.cc:302] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8248e297a6b448f99ec43c55d912ae5f, 91100cd166fe4806b89e404940eeae8b; no voters:
I20260927 15:03:58.700405 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.696065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41771 (local address 127.0.98.3:42265)
0927 15:03:58.696486 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.696502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:03:58.696583 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:03:58.696971 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:03:58.696973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:03:58.696990 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:03:58.697063 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.697068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.698055 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.698058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.699119 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:03:58.699129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.700070 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.700073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:03:58.700151 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.700172 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:03:58.700236 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:03:58.700294 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355}
I20260927 15:03:58.700773 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.696284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42265 (local address 127.0.98.2:41771)
0927 15:03:58.696402 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.696406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.696664 (+ 258us) server_negotiation.cc:407] Connection header received
0927 15:03:58.696720 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.696722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.696773 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.696858 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:03:58.697188 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.697932 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.699257 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.699855 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.699955 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.700014 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.700562 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.700645 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:03:58.701115 12130 leader_election.cc:288] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91100cd166fe4806b89e404940eeae8b (127.0.98.1:35679), 970184cee94442c782e4349f96d15d98 (127.0.98.2:41771)
I20260927 15:03:58.701292 12130 raft_consensus.cc:2781] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:03:58.701452 12130 raft_consensus.cc:482] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:03:58.701568 12130 raft_consensus.cc:3037] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:58.702441 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b9ac2fb0af4779aaa561152b883e70" candidate_uuid: "8248e297a6b448f99ec43c55d912ae5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970184cee94442c782e4349f96d15d98" is_pre_election: true
I20260927 15:03:58.702747 11910 raft_consensus.cc:2454] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8248e297a6b448f99ec43c55d912ae5f in term 0.
I20260927 15:03:58.703091 12130 raft_consensus.cc:504] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.703441 12130 leader_election.cc:288] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [CANDIDATE]: Term 1 election: Requested vote from peers 91100cd166fe4806b89e404940eeae8b (127.0.98.1:35679), 970184cee94442c782e4349f96d15d98 (127.0.98.2:41771)
I20260927 15:03:58.704110 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b9ac2fb0af4779aaa561152b883e70" candidate_uuid: "8248e297a6b448f99ec43c55d912ae5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970184cee94442c782e4349f96d15d98"
I20260927 15:03:58.704241 11910 raft_consensus.cc:3037] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:58.705473 11910 raft_consensus.cc:2454] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8248e297a6b448f99ec43c55d912ae5f in term 1.
I20260927 15:03:58.705849 11993 leader_election.cc:302] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8248e297a6b448f99ec43c55d912ae5f, 970184cee94442c782e4349f96d15d98; no voters:
I20260927 15:03:58.706040 12130 raft_consensus.cc:2781] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:03:58.706135 12130 raft_consensus.cc:686] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 1 LEADER]: Becoming Leader. State: Replica: 8248e297a6b448f99ec43c55d912ae5f, State: Running, Role: LEADER
I20260927 15:03:58.706481 12130 consensus_queue.cc:237] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } }
I20260927 15:03:58.709903 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b9ac2fb0af4779aaa561152b883e70" candidate_uuid: "8248e297a6b448f99ec43c55d912ae5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91100cd166fe4806b89e404940eeae8b"
I20260927 15:03:58.710040 11787 raft_consensus.cc:3037] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:03:58.711262 11787 raft_consensus.cc:2454] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8248e297a6b448f99ec43c55d912ae5f in term 1.
I20260927 15:03:58.711552 11620 catalog_manager.cc:5495] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8248e297a6b448f99ec43c55d912ae5f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8248e297a6b448f99ec43c55d912ae5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8248e297a6b448f99ec43c55d912ae5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44745 } health_report { overall_health: HEALTHY } } }
Sep 27 15:03:58 dist-test-slave-ml7t krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:03:58.756436 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.750923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34730 (local address 127.0.98.3:44745)
0927 15:03:58.751046 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:03:58.751050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:03:58.751064 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:03:58.751151 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:03:58.751153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:03:58.751219 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:03:58.751329 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:03:58.751733 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.752559 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.753921 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:03:58.754237 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:03:58.754339 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:03:58.754762 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:03:58.755405 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:03:58.755529 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:03:58.756208 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 15:03:58.756269 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 15:03:58.763892 12106 tablet.cc:2369] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:03:58.768558 11910 raft_consensus.cc:1261] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 1 FOLLOWER]: Refusing update from remote peer 8248e297a6b448f99ec43c55d912ae5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:03:58.768611 11787 raft_consensus.cc:1261] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b [term 1 FOLLOWER]: Refusing update from remote peer 8248e297a6b448f99ec43c55d912ae5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:03:58.769345 12130 consensus_queue.cc:1035] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "970184cee94442c782e4349f96d15d98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:03:58.769963 12133 consensus_queue.cc:1035] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91100cd166fe4806b89e404940eeae8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:03:58.794379 11836 tablet.cc:2369] T 82b9ac2fb0af4779aaa561152b883e70 P 91100cd166fe4806b89e404940eeae8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:03:58.826431 12139 mvcc.cc:205] Tried to move back new op lower bound from 7333975813185347584 to 7333975812947140608. Current Snapshot: MvccSnapshot[applied={T|T < 7333975813185347584 or (T in {7333975813185347584})}]
I20260927 15:03:58.830554 11620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:03:58.830739 11620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:03:58.834228 11620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda69f92286f4c46a1a8499fbba6ea7d: Sending DeleteTablet for 3 replicas of tablet 82b9ac2fb0af4779aaa561152b883e70
I20260927 15:03:58.834911 11764 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b9ac2fb0af4779aaa561152b883e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43468
I20260927 15:03:58.835845 11905 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b9ac2fb0af4779aaa561152b883e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34048
I20260927 15:03:58.836697 12039 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b9ac2fb0af4779aaa561152b883e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34722
I20260927 15:03:58.838052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 11702
I20260927 15:03:58.845989 12150 tablet_replica.cc:330] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: stopping tablet replica
I20260927 15:03:58.846391 12150 raft_consensus.cc:2229] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:03:58.846828 12150 raft_consensus.cc:2258] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:03:58.848569 12150 ts_tablet_manager.cc:1905] T 82b9ac2fb0af4779aaa561152b883e70 P 8248e297a6b448f99ec43c55d912ae5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:03:58.850037 12149 tablet_replica.cc:330] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98: stopping tablet replica
I20260927 15:03:58.850431 12149 raft_consensus.cc:2229] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 15:03:58.850487 11606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35679 (error 108)
I20260927 15:03:58.850848 12149 raft_consensus.cc:2258] T 82b9ac2fb0af4779aaa561152b883e70 P 970184cee94442c782e4349f96d15d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:03:58.851358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 11841
I20260927 15:03:58.855458 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.855092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35679 (local address 127.0.0.1:43482)
0927 15:03:58.855259 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.855346 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:03:58.855635 11606 catalog_manager.cc:4584] TS 91100cd166fe4806b89e404940eeae8b (127.0.98.1:35679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82b9ac2fb0af4779aaa561152b883e70: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35679: connect: Connection refused (error 111)
I20260927 15:03:58.860131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 11976
I20260927 15:03:58.860378 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:03:58.860059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41771 (local address 127.0.0.1:34056)
0927 15:03:58.860246 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 15:03:58.860299 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 15:03:58.868237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 11589
I20260927 15:03:58.875916 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:03:59.220439 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:03:59.220593 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:03:59.565632 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:03:59.565 UTC [11175] LOG: received smart shutdown request
2026-09-27 15:03:59.569 UTC [11175] LOG: background worker "logical replication launcher" (PID 11184) exited with exit code 1
2026-09-27 15:03:59.570 UTC [11179] LOG: shutting down
2026-09-27 15:03:59.734 UTC [11175] LOG: database system is shut down
2026-09-27T15:03:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47198 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:03:59 dist-test-slave-ml7t krb5kdc[12160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:03:59 dist-test-slave-ml7t krb5kdc[12160](info): set up 2 sockets
Sep 27 15:03:59 dist-test-slave-ml7t krb5kdc[12160](info): commencing operation
krb5kdc: starting...
W20260927 15:04:00.901846 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:04:00 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:04:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 15:04:01.074713 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYaV7G8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/postgres -l logfile start
2026-09-27 15:04:03.590 UTC [12190] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:04:03.590 UTC [12190] LOG: listening on IPv4 address "127.0.98.20", port 53887
2026-09-27 15:04:03.594 UTC [12190] LOG: listening on Unix socket "/tmp/.s.PGSQL.53887"
2026-09-27 15:04:03.615 UTC [12193] LOG: database system was shut down at 2026-09-27 15:04:02 UTC
2026-09-27 15:04:03.618 UTC [12190] LOG: database system is ready to accept connections
I20260927 15:04:04.577734 392 mini_postgres.cc:96] Postgres bound to 53887
2026-09-27 15:04:04.584 UTC [12201] FATAL: database "slave" does not exist
127.0.98.20:53887 - accepting connections
I20260927 15:04:04.584976 392 mini_ranger.cc:161] Starting Ranger...
I20260927 15:04:04.599802 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:04:05.077986 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:04:05.078083 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-admin
2026-09-27 15:04:05,350 [I] DB FLAVOR :POSTGRES
2026-09-27 15:04:05,351 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:04:05,351 [I] Checking connection..
2026-09-27 15:04:05,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:04:05,598 [I] Checking connection passed.
2026-09-27 15:04:05,599 [I] --------- Verifying version history table ---------
2026-09-27 15:04:05,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:04:05,831 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:04:05,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:04:06,063 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:04:06,063 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:04:06,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:04:06.312 UTC [12265] WARNING: there is no transaction in progress
2026-09-27 15:04:06,318 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:04:06,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:04:06,574 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:04:06,575 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:04:06,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:04:06,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:04:07,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ml7t.c.gcp-upstream.internal','N') ;"
2026-09-27 15:04:07,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:04:07,541 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:04:07,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:04:07,801 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:04:07,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:04:08,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:04:08,281 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:04:08,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:04:09.251 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.266 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.283 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.299 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.315 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.332 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.386 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.393 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.407 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.417 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.427 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.441 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.451 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.458 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.465 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:09.472 UTC [12373] WARNING: there is no transaction in progress
2026-09-27 15:04:10,141 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:04:10,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:04:10,416 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:04:10,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:04:10,663 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:04:10,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:04:10,898 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:04:10,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:04:11,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:04:11,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ml7t.c.gcp-upstream.internal';"
2026-09-27 15:04:11,607 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:04:11,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:04:11,877 [I] DB_PATCHES have already been applied
I20260927 15:04:11.882715 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYaV7G8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:04:11.882795 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 15:04:11.886304 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:38429
Sep 27, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:04:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:04:13.006726 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.422s user 0.001s sys 0.009s
Sep 27, 2026 3:04:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:04:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:04:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:04:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:04:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:04:35.584113 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:04:35 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521475, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:04:35.670517 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:04:35.679827 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:04:36.561405 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:04:36.561481 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms
I20260927 15:04:36.561530 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53887
2026-09-27 15:04:36,730 [I] DB FLAVOR :POSTGRES
2026-09-27 15:04:36,730 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:04:36,731 [I] Checking connection
2026-09-27 15:04:36,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:04:36,977 [I] connection success
2026-09-27 15:04:36,977 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:04:36,977 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:04:36,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:04:37,230 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:04:37,230 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:04:37,230 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:04:37,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53887/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:04:37,496 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:04:37.955175 392 mini_ranger_kms.cc:310] Created kms service
I20260927 15:04:38.134322 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:04:38.179106 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:04:38 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521475, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:38.578684 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:04:38.578862 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYaV7G8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYaV7G8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:04:38.578893 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 15:04:38.582885 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12553
I20260927 15:04:38.582975 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36093
Sep 27, 2026 3:04:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:04:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:04:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:04:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:04:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:04:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:04:40.926048 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.256s user 0.005s sys 0.144s
I20260927 15:04:40.926175 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:04:40.926273 392 mini_ranger_kms.cc:401] 127.0.98.20:36093/kms/v1/keys
Sep 27 15:04:40 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521480, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:04:41 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521480, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 15:04:41 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521480, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 15:04:43 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:04:44.016366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46675
--ranger_config_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:44.142881 12604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:44.143133 12604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:44.143172 12604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:44.147112 12604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:44.147167 12604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:44.147187 12604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:44.147203 12604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:44.147217 12604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:44.152806 12604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46675
--ranger_config_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:44.153810 12604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:44.154796 12604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:44.163501 12610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:44.163614 12612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:44.163628 12604 server_base.cc:1034] running on GCE node
W20260927 15:04:44.163770 12609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:44.164325 12604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:44.165408 12604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:44.166621 12604 hybrid_clock.cc:648] HybridClock initialized: now 1790521484166596 us; error 44 us; skew 500 ppm
Sep 27 15:04:44 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:44.169472 12604 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:04:44.170492 12604 webserver.cc:460] Webserver started at http://127.0.98.62:34691/ using document root <none> and password file <none>
I20260927 15:04:44.171085 12604 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:44.171151 12604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:44.171370 12604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:04:44 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:44.218981 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "ddc2d8ef96c1410f81084c927eb99b8f"
format_stamp: "Formatted at 2026-09-27 15:04:44 on dist-test-slave-ml7t"
server_key: "b50935af43c1b015db0bba1a883162b7"
server_key_iv: "5fcb19289b9d8b2069168d4601736a69"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:44.219709 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "ddc2d8ef96c1410f81084c927eb99b8f"
format_stamp: "Formatted at 2026-09-27 15:04:44 on dist-test-slave-ml7t"
server_key: "b50935af43c1b015db0bba1a883162b7"
server_key_iv: "5fcb19289b9d8b2069168d4601736a69"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:44.224764 12604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:04:44.240399 12619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:44.241941 12604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:04:44.242080 12604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "ddc2d8ef96c1410f81084c927eb99b8f"
format_stamp: "Formatted at 2026-09-27 15:04:44 on dist-test-slave-ml7t"
server_key: "b50935af43c1b015db0bba1a883162b7"
server_key_iv: "5fcb19289b9d8b2069168d4601736a69"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:44.242235 12604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:44.255129 12604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:44.258355 12604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:44.258562 12604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:44.266424 12604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46675
I20260927 15:04:44.266434 12671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46675 every 8 connection(s)
I20260927 15:04:44.267390 12604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:04:44.270807 12672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:44.274471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 12604
I20260927 15:04:44.274626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:04:44.274904 392 external_mini_cluster.cc:1428] Setting key 9f231f8569eb9a3ff1219030a21b489d
I20260927 15:04:44.279469 12672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Bootstrap starting.
I20260927 15:04:44.282253 12672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:44.283166 12672 log.cc:824] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:04:44 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:44.285538 12672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: No bootstrap required, opened a new log
I20260927 15:04:44.289605 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:44.276294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.0.98.62:46675)
0927 15:04:44.276728 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:44.276744 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:44.276778 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:04:44.277425 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:44.277441 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:44.277822 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:44.278215 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:44.279008 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:44.280081 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:44.280962 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:44.281442 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:44.281619 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:44.284382 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:44.284388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:44.284402 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:44.284427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:44.286768 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:44.287255 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:44.287261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:44.287274 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:44.287379 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:44.287687 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:44.287690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:44.287692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:44.288091 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:44.288298 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:44.288569 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:04:44.288867 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260927 15:04:44.289660 12672 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:44.290946 12672 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:44.291054 12672 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc2d8ef96c1410f81084c927eb99b8f, State: Initialized, Role: FOLLOWER
I20260927 15:04:44.291595 12672 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:44.291852 12672 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:04:44.291980 12672 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:04:44.292091 12672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:44.293316 12672 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:44.293673 12672 leader_election.cc:302] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddc2d8ef96c1410f81084c927eb99b8f; no voters:
I20260927 15:04:44.294065 12672 leader_election.cc:288] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:04:44.294771 12677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:04:44.295186 12677 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 LEADER]: Becoming Leader. State: Replica: ddc2d8ef96c1410f81084c927eb99b8f, State: Running, Role: LEADER
I20260927 15:04:44.295397 12672 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:44.295544 12677 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:44.297240 12679 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddc2d8ef96c1410f81084c927eb99b8f. Latest consensus state: current_term: 1 leader_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } }
I20260927 15:04:44.297369 12679 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:44.297665 12672 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:04:44.297675 12678 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } }
I20260927 15:04:44.298251 12678 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:44.299017 12691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:04:44.304324 12691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:04:44.311681 12691 catalog_manager.cc:1347] Generated new cluster ID: bec739a4b7c3436d804fbe4e2ba309aa
I20260927 15:04:44.311790 12691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:04:44.341506 12691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:04:44.343019 12691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:04:44.354318 12691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Generated new TSK 0
I20260927 15:04:44.355234 12691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:04:45.029989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46675
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:45.176404 12717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:45.176764 12717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:45.176827 12717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:45.182926 12717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:45.183012 12717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:45.183102 12717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:04:45.190897 12717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:45.192270 12717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:45.193581 12717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:45.205641 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:45.205633 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:45.205637 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:45.206274 12717 server_base.cc:1034] running on GCE node
I20260927 15:04:45.206687 12717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:45.207341 12717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:45.208540 12717 hybrid_clock.cc:648] HybridClock initialized: now 1790521485208515 us; error 40 us; skew 500 ppm
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:45.212332 12717 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:04:45.213433 12717 webserver.cc:460] Webserver started at http://127.0.98.1:37641/ using document root <none> and password file <none>
I20260927 15:04:45.214147 12717 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:45.214233 12717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:45.214507 12717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:45.247198 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "9668d4d52c684b5c9e52b95e8d11ecae"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "80e64777cf1a25f3baa7013ca4936e18"
server_key_iv: "e2b2681cc64d568f948435a748b130f2"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.247936 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9668d4d52c684b5c9e52b95e8d11ecae"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "80e64777cf1a25f3baa7013ca4936e18"
server_key_iv: "e2b2681cc64d568f948435a748b130f2"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.252816 12717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:04:45.278105 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.283285 12717 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.002s
I20260927 15:04:45.283483 12717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "9668d4d52c684b5c9e52b95e8d11ecae"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "80e64777cf1a25f3baa7013ca4936e18"
server_key_iv: "e2b2681cc64d568f948435a748b130f2"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.283622 12717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:45.318068 12717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:45.321488 12717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:45.321704 12717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:45.322342 12717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:45.323236 12717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:45.323304 12717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.323390 12717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:45.323447 12717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.334554 12717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41665
I20260927 15:04:45.334578 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41665 every 8 connection(s)
I20260927 15:04:45.335564 12717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:04:45.339320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 12717
I20260927 15:04:45.339445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:04:45.339758 392 external_mini_cluster.cc:1428] Setting key aacc6d5de5300fd9908d2b168eb94432
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:04:45.352399 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:45.337492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60529 (local address 127.0.98.62:46675)
0927 15:04:45.337811 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:45.337820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:45.338596 (+ 776us) server_negotiation.cc:407] Connection header received
0927 15:04:45.339858 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:45.339861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:45.339922 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:45.339989 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:45.341551 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.342427 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.343880 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.344289 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.344399 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:45.347767 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:45.347773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:45.347777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:45.347811 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:45.349853 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:45.350479 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:45.350485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:45.350489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:45.350581 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:45.351034 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:45.351039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:45.351041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:45.351262 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:45.351352 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:45.352071 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 15:04:45.352215 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
I20260927 15:04:45.353302 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:45.337988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.98.1:60529)
0927 15:04:45.338443 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:45.338495 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:45.339581 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:45.340387 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:45.340394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:45.340735 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:45.341386 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:45.341408 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.342591 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.342597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:45.343755 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:45.343765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.344579 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.344585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:45.344805 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:45.345767 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:45.345813 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:45.347576 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:45.349993 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:45.350000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:45.350015 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:45.350361 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:45.350713 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:45.350718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:45.350721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:45.350908 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:45.351408 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:45.351417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:45.351784 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 15:04:45.352019 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 15:04:45.352338 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:04:45.354846 12850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46675
I20260927 15:04:45.355070 12850 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:45.355664 12850 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:45.357460 12636 ts_manager.cc:194] Registered new tserver with Master: 9668d4d52c684b5c9e52b95e8d11ecae (127.0.98.1:41665)
I20260927 15:04:45.358625 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60529
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:04:45.372766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46675
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:45.497944 12856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:45.498359 12856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:45.498477 12856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:45.503430 12856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:45.503522 12856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:45.503633 12856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:04:45.509447 12856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:45.510802 12856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:45.512506 12856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:45.523000 12863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:45.523301 12865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:45.523989 12862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:45.524317 12856 server_base.cc:1034] running on GCE node
I20260927 15:04:45.524718 12856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:45.525421 12856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:45.526602 12856 hybrid_clock.cc:648] HybridClock initialized: now 1790521485526565 us; error 47 us; skew 500 ppm
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:45.530030 12856 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:04:45.531082 12856 webserver.cc:460] Webserver started at http://127.0.98.2:32829/ using document root <none> and password file <none>
I20260927 15:04:45.531869 12856 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:45.532015 12856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:45.532302 12856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:45.546075 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee9f757a2fc9487d9d45af47d0eead33"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "5b83281acd7d0ca4e25c49d44ed0595e"
server_key_iv: "e2ac8eb396aee2875ac993d1b63f50f4"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.547155 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee9f757a2fc9487d9d45af47d0eead33"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "5b83281acd7d0ca4e25c49d44ed0595e"
server_key_iv: "e2ac8eb396aee2875ac993d1b63f50f4"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.554083 12856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 15:04:45.569643 12872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.571117 12856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:04:45.571270 12856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "ee9f757a2fc9487d9d45af47d0eead33"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "5b83281acd7d0ca4e25c49d44ed0595e"
server_key_iv: "e2ac8eb396aee2875ac993d1b63f50f4"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.571416 12856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:45.582749 12856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:45.586239 12856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:45.586498 12856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:45.587386 12856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:45.588459 12856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:45.588564 12856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.588757 12856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:45.588840 12856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.601408 12856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38095
I20260927 15:04:45.601470 12985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38095 every 8 connection(s)
I20260927 15:04:45.602748 12856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:04:45.604439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 12856
I20260927 15:04:45.604565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:04:45.604877 392 external_mini_cluster.cc:1428] Setting key 71a90230e757268ec87663fe64fa7374
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:04:45.622504 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:45.605949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38823 (local address 127.0.98.62:46675)
0927 15:04:45.606351 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:45.606356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:45.606977 (+ 621us) server_negotiation.cc:407] Connection header received
0927 15:04:45.608182 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:45.608186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:45.608235 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:45.608289 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:45.609894 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.610712 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.612706 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.613074 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.613155 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:45.616120 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:45.616124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:45.616127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:45.616154 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:45.617957 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:45.620555 (+ 2598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:45.620559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:45.620561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:45.620656 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:45.621088 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:45.621092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:45.621093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:45.621339 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:45.621435 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:45.622195 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 15:04:45.622330 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260927 15:04:45.623543 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:45.605438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.98.2:38823)
0927 15:04:45.606834 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:45.606888 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:45.607929 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:45.608684 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:45.608693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:45.609056 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:45.609701 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:45.609720 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.610870 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.610873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:45.612265 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:45.612276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.613349 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.613353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:45.613542 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:45.614223 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:45.614243 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:45.615922 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:45.618089 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:45.618095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:45.618109 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:45.618592 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:45.620781 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:45.620784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:45.620786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:45.620970 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:45.621508 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:45.621514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:45.621890 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 15:04:45.622110 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 15:04:45.622469 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":116,"threads_started":1}
I20260927 15:04:45.625164 12986 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46675
I20260927 15:04:45.625430 12986 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:45.626061 12986 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:45.627286 12636 ts_manager.cc:194] Registered new tserver with Master: ee9f757a2fc9487d9d45af47d0eead33 (127.0.98.2:38095)
I20260927 15:04:45.628103 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:04:45.640714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46675
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:45.794152 12991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:45.794574 12991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:45.794690 12991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 15:04:45.801910 12991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:45.802035 12991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:45.802155 12991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:04:45.810004 12991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:45.811641 12991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:45.813059 12991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:45.826790 12996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:45.827265 12997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:45.827512 12999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:45.828702 12991 server_base.cc:1034] running on GCE node
I20260927 15:04:45.829226 12991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:45.830075 12991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:45.831318 12991 hybrid_clock.cc:648] HybridClock initialized: now 1790521485831256 us; error 78 us; skew 500 ppm
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:45.835158 12991 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:04:45.836189 12991 webserver.cc:460] Webserver started at http://127.0.98.3:33743/ using document root <none> and password file <none>
I20260927 15:04:45.837074 12991 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:45.837222 12991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:45.837512 12991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:45.865149 12991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "1b90e7340ed84b1f9fc1d57f7572c796"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "fcfce661779f85561be1264ead20bdca"
server_key_iv: "29c16097bc5f919d880f4f1cc504454d"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.865980 12991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1b90e7340ed84b1f9fc1d57f7572c796"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "fcfce661779f85561be1264ead20bdca"
server_key_iv: "29c16097bc5f919d880f4f1cc504454d"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.880891 12991 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260927 15:04:45.902014 13006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.903860 12991 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 15:04:45.904006 12991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "1b90e7340ed84b1f9fc1d57f7572c796"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "fcfce661779f85561be1264ead20bdca"
server_key_iv: "29c16097bc5f919d880f4f1cc504454d"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:45.904141 12991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:45.922657 12991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:45.926499 12991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:45.926700 12991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:45.927393 12991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:45.928496 12991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:45.928566 12991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:45.928627 12991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:45.928717 12991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 15:04:45.941627 12991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37583
I20260927 15:04:45.941902 13119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37583 every 8 connection(s)
I20260927 15:04:45.942839 12991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:04:45.950780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 12991
I20260927 15:04:45.950933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:04:45.951247 392 external_mini_cluster.cc:1428] Setting key d6d6cc4b5db5af7c31cb0c64870a97e0
Sep 27 15:04:45 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:45.958055 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:45.944819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59613 (local address 127.0.98.62:46675)
0927 15:04:45.944962 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:45.944967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:45.946069 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 15:04:45.947185 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:45.947189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:45.947252 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:45.947363 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:45.948484 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.949036 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.950324 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.950692 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.950789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:45.953997 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:45.954000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:45.954003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:45.954027 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:45.955774 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:45.956371 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:45.956375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:45.956377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:45.956468 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:45.956848 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:45.956851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:45.956853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:45.957059 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:45.957171 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:45.957708 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 15:04:45.957860 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:04:45.958936 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:45.945355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.98.3:59613)
0927 15:04:45.945856 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:45.945911 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:45.946954 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:45.947547 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:45.947554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:45.947831 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:45.948327 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:45.948342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.949169 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.949172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:45.950151 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:45.950161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:45.951053 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:45.951056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:45.951211 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:45.951920 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:45.951943 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:45.953791 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:45.955925 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:45.955931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:45.955949 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:45.956236 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:45.956575 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:45.956580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:45.956583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:45.956733 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:45.957179 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:45.957189 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:45.957447 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 15:04:45.957650 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 15:04:45.958001 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":141,"threads_started":1}
I20260927 15:04:45.960140 13120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46675
I20260927 15:04:45.960403 13120 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:45.960906 13120 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:45.962021 12636 ts_manager.cc:194] Registered new tserver with Master: 1b90e7340ed84b1f9fc1d57f7572c796 (127.0.98.3:37583)
I20260927 15:04:45.962631 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59613
I20260927 15:04:45.965523 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:04:45.966315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 12717
I20260927 15:04:45.972929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 12856
I20260927 15:04:45.979535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 12991
I20260927 15:04:45.985751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 12604
I20260927 15:04:45.995553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--webserver_interface=127.0.98.62
--webserver_port=34691
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46675
--ranger_config_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:46.107620 13123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:46.107883 13123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:46.107923 13123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:46.111974 13123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:46.112032 13123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:46.112051 13123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:46.112068 13123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:46.112105 13123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:46.117487 13123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46675
--ranger_config_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:46.118541 13123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:46.119858 13123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:46.128255 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.128264 13129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.128266 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:46.128533 13123 server_base.cc:1034] running on GCE node
I20260927 15:04:46.129141 13123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:46.130277 13123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:46.131430 13123 hybrid_clock.cc:648] HybridClock initialized: now 1790521486131396 us; error 46 us; skew 500 ppm
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:46.134305 13123 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:04:46.135186 13123 webserver.cc:460] Webserver started at http://127.0.98.62:34691/ using document root <none> and password file <none>
I20260927 15:04:46.135670 13123 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:46.135732 13123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:46.182011 13123 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:04:46.184247 13139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.185613 13123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:04:46.185812 13123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "ddc2d8ef96c1410f81084c927eb99b8f"
format_stamp: "Formatted at 2026-09-27 15:04:44 on dist-test-slave-ml7t"
server_key: "b50935af43c1b015db0bba1a883162b7"
server_key_iv: "5fcb19289b9d8b2069168d4601736a69"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:46.186380 13123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:46.207193 13123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:46.209806 13123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:46.210000 13123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:46.216898 13123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46675
I20260927 15:04:46.216905 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46675 every 8 connection(s)
I20260927 15:04:46.217940 13123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:04:46.221995 13192 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:04:46.222730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 13123
I20260927 15:04:46.223155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41665
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46675
--webserver_port=37641
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:46.223668 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Bootstrap starting.
I20260927 15:04:46.239734 13192 log.cc:824] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:46.246172 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:04:46.246582 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Bootstrap complete.
I20260927 15:04:46.250641 13192 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:46.251046 13192 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc2d8ef96c1410f81084c927eb99b8f, State: Initialized, Role: FOLLOWER
I20260927 15:04:46.251605 13192 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:46.251753 13192 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:04:46.251842 13192 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:04:46.251955 13192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:04:46.253804 13192 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:46.254179 13192 leader_election.cc:302] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddc2d8ef96c1410f81084c927eb99b8f; no voters:
I20260927 15:04:46.254557 13192 leader_election.cc:288] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:04:46.254642 13196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:04:46.254916 13196 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [term 2 LEADER]: Becoming Leader. State: Replica: ddc2d8ef96c1410f81084c927eb99b8f, State: Running, Role: LEADER
I20260927 15:04:46.255278 13196 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } }
I20260927 15:04:46.255820 13192 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:46.256691 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } }
I20260927 15:04:46.256810 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:46.257020 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddc2d8ef96c1410f81084c927eb99b8f. Latest consensus state: current_term: 2 leader_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc2d8ef96c1410f81084c927eb99b8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } }
I20260927 15:04:46.257105 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:46.257675 13205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:04:46.258193 13192 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:04:46.263820 13205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:04:46.264907 13205 catalog_manager.cc:1259] Loaded cluster ID: bec739a4b7c3436d804fbe4e2ba309aa
I20260927 15:04:46.265025 13205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:04:46.268544 13205 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:04:46.269343 13205 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Loaded TSK: 0
I20260927 15:04:46.270853 13205 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:04:46.372354 13194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:46.372637 13194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:46.372737 13194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:46.377431 13194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:46.377511 13194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:46.377635 13194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:04:46.383736 13194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37641
--tserver_master_addrs=127.0.98.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:46.384810 13194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:46.385989 13194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:46.398480 13229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.398607 13232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.398484 13230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:46.399488 13194 server_base.cc:1034] running on GCE node
I20260927 15:04:46.399969 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:46.400748 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:46.404049 13194 hybrid_clock.cc:648] HybridClock initialized: now 1790521486404017 us; error 38 us; skew 500 ppm
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:46.407549 13194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:04:46.408578 13194 webserver.cc:460] Webserver started at http://127.0.98.1:37641/ using document root <none> and password file <none>
I20260927 15:04:46.409296 13194 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:46.409441 13194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:46.440078 13194 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:04:46.445996 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.447528 13194 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 15:04:46.447661 13194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "9668d4d52c684b5c9e52b95e8d11ecae"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "80e64777cf1a25f3baa7013ca4936e18"
server_key_iv: "e2b2681cc64d568f948435a748b130f2"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:46.448393 13194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:46.480788 13194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:46.484660 13194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:46.484896 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:46.485652 13194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:46.486635 13194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:46.486709 13194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.486774 13194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:46.486800 13194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.497531 13194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41665
I20260927 15:04:46.498883 13194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:04:46.499634 13353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41665 every 8 connection(s)
I20260927 15:04:46.502079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 13194
I20260927 15:04:46.502578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38095
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46675
--webserver_port=32829
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:46.531706 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:46.501719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.98.1:49027)
0927 15:04:46.503157 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:46.503216 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:46.504333 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:46.508890 (+ 4557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:46.508900 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:46.509238 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:46.509962 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:46.509984 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.513916 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.513920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:46.515309 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:46.515321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.516406 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.516411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:46.516623 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:46.517385 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:46.517405 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:46.518956 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:46.527772 (+ 8816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:46.527779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:46.527795 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:46.528185 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:46.528654 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:46.528658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:46.528660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:46.528848 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:46.529654 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:46.529661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:46.530025 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 15:04:46.530212 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 15:04:46.530560 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":129,"threads_started":1}
I20260927 15:04:46.537515 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:46.504361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49027 (local address 127.0.98.62:46675)
0927 15:04:46.507564 (+ 3203us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:46.507588 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:46.507637 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:04:46.507953 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:46.507969 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:46.508384 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:46.510199 (+ 1815us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:46.510427 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.511779 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.515475 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.516215 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.517032 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:46.524001 (+ 6969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:46.524008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:46.524029 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:46.524064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:46.527612 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:46.528359 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:46.528365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:46.528371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:46.528517 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:46.528970 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:46.528973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:46.528975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:46.529479 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:46.536084 (+ 6605us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:46.536264 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 15:04:46.536556 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3043,"thread_start_us":111,"threads_started":1}
I20260927 15:04:46.540453 13354 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46675
I20260927 15:04:46.540683 13354 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:46.541217 13354 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
W20260927 15:04:46.543428 13354 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:04:46.543540 13354 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:46.543694 13354 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:46.544629 13354 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:46.544781 13354 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
W20260927 15:04:46.546728 13354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:04:46.646603 13357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:46.646896 13357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:46.646973 13357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:46.651252 13357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:46.651343 13357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:46.651436 13357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:04:46.657881 13357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32829
--tserver_master_addrs=127.0.98.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:46.659220 13357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:46.660511 13357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:46.674029 13366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.674409 13363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.674906 13364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:46.675482 13357 server_base.cc:1034] running on GCE node
I20260927 15:04:46.676019 13357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:46.676745 13357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:46.680055 13357 hybrid_clock.cc:648] HybridClock initialized: now 1790521486679997 us; error 49 us; skew 500 ppm
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:46.682955 13357 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:04:46.684017 13357 webserver.cc:460] Webserver started at http://127.0.98.2:32829/ using document root <none> and password file <none>
I20260927 15:04:46.684732 13357 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:46.684827 13357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:46.707885 13357 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:04:46.711071 13374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.712746 13357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:04:46.712874 13357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "ee9f757a2fc9487d9d45af47d0eead33"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "5b83281acd7d0ca4e25c49d44ed0595e"
server_key_iv: "e2ac8eb396aee2875ac993d1b63f50f4"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:46.713616 13357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:46.748329 13357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:46.752252 13357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:46.752480 13357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:46.753342 13357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:46.754474 13357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:46.754536 13357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.754586 13357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:46.754616 13357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.766330 13357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38095
I20260927 15:04:46.767702 13357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:04:46.768632 13488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38095 every 8 connection(s)
I20260927 15:04:46.772467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 13357
I20260927 15:04:46.772991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37583
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46675
--webserver_port=33743
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:46.784893 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:46.770376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52301 (local address 127.0.98.62:46675)
0927 15:04:46.770547 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:46.770552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:46.771459 (+ 907us) server_negotiation.cc:407] Connection header received
0927 15:04:46.772807 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:46.772810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:46.772865 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:46.772969 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:46.774459 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.775053 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.776317 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.776582 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.776674 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:46.779859 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:46.779866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:46.779872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:46.779909 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:46.781937 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:46.782657 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:46.782662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:46.782666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:46.782778 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:46.783260 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:46.783267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:46.783272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:46.783548 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:46.783635 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:46.784471 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 15:04:46.784634 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:04:46.787137 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:46.770835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.98.2:52301)
0927 15:04:46.771293 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:46.771343 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:46.772558 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:46.773177 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:46.773188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:46.773540 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:46.774263 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:46.774285 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.775181 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.775185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:46.776178 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:46.776188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:46.776889 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:46.776895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:46.777121 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:46.777929 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:46.777953 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:46.779633 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:46.782094 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:46.782103 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:46.782121 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:46.782524 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:46.782909 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:46.782912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:46.782917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:46.783122 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:46.783746 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:46.783754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:46.784142 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 15:04:46.785429 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0927 15:04:46.785890 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":128,"threads_started":1}
I20260927 15:04:46.788864 13489 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46675
I20260927 15:04:46.789136 13489 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:46.789800 13489 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
W20260927 15:04:46.791549 13489 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:04:46.791682 13489 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:46.791879 13489 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:46.792892 13489 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:46.793073 13489 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
W20260927 15:04:46.794031 13489 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:04:46.911747 13492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:46.912091 13492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:46.912171 13492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:46.917608 13492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:46.917693 13492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:46.917881 13492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:04:46.923933 13492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33743
--tserver_master_addrs=127.0.98.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:46.925087 13492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:46.926297 13492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:46.937598 13498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.937645 13497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:46.937585 13500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:46.938234 13492 server_base.cc:1034] running on GCE node
I20260927 15:04:46.938645 13492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:46.939288 13492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:46.940472 13492 hybrid_clock.cc:648] HybridClock initialized: now 1790521486940440 us; error 51 us; skew 500 ppm
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:46.943545 13492 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:04:46.944646 13492 webserver.cc:460] Webserver started at http://127.0.98.3:33743/ using document root <none> and password file <none>
I20260927 15:04:46.945322 13492 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:46.945422 13492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:04:46 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:46.965391 13492 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.002s
I20260927 15:04:46.968235 13508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:46.969559 13492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:04:46.969700 13492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "1b90e7340ed84b1f9fc1d57f7572c796"
format_stamp: "Formatted at 2026-09-27 15:04:45 on dist-test-slave-ml7t"
server_key: "fcfce661779f85561be1264ead20bdca"
server_key_iv: "29c16097bc5f919d880f4f1cc504454d"
server_key_version: "kuduclusterkey@0"
I20260927 15:04:46.970304 13492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:47.001859 13492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:47.005527 13492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:47.005748 13492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:47.006453 13492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:47.007507 13492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:47.007573 13492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:47.007642 13492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:47.007673 13492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:47.019177 13492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37583
I20260927 15:04:47.019243 13628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37583 every 8 connection(s)
I20260927 15:04:47.020233 13492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:04:47.022065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 13492
Sep 27 15:04:47 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:47.035097 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.023576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49861 (local address 127.0.98.62:46675)
0927 15:04:47.023731 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.023735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.023761 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:04:47.024417 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.024420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.024474 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.024571 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:47.025957 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.026501 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.028129 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.028349 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.028460 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.031110 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:47.031113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:47.031116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:47.031140 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:47.032776 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:47.033375 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:47.033379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:47.033381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:47.033467 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:47.033869 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:47.033872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:47.033874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:47.034064 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:47.034176 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.034800 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.034925 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":71}
I20260927 15:04:47.036383 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.022643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.98.3:49861)
0927 15:04:47.023106 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:47.023156 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:47.024192 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:47.024704 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:47.024711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:47.025079 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:47.025813 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.025833 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.026629 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.026632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.027995 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.028003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.028657 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.028660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.028807 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.029475 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:47.029492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:47.030917 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:47.032908 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:47.032914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:47.032931 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:47.033267 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:47.033568 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:47.033571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:47.033573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:47.033718 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:47.034174 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:47.034181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:47.034484 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 15:04:47.034694 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 15:04:47.035016 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":132,"threads_started":1}
I20260927 15:04:47.037652 13629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46675
I20260927 15:04:47.037953 13629 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:47.038511 13629 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:47.039806 13156 ts_manager.cc:194] Registered new tserver with Master: 1b90e7340ed84b1f9fc1d57f7572c796 (127.0.98.3:37583)
I20260927 15:04:47.040904 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49861
I20260927 15:04:47.041347 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.023676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.0.98.62:46675)
0927 15:04:47.023999 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.024007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.024306 (+ 299us) server_negotiation.cc:407] Connection header received
0927 15:04:47.024678 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.024681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.024712 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.024797 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:47.025720 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.026609 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.034337 (+ 7728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.034735 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.034847 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.037108 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:47.037112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:47.037115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:47.037145 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:47.038962 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:47.039453 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:47.039457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:47.039459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:47.039550 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:47.039874 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:47.039877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:47.039879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:47.040086 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:47.040243 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.041038 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.041168 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
Sep 27 15:04:47 dist-test-slave-ml7t krb5kdc[12160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:47.546986 13354 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:47.547304 13354 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:47.548213 13156 ts_manager.cc:194] Registered new tserver with Master: 9668d4d52c684b5c9e52b95e8d11ecae (127.0.98.1:41665)
I20260927 15:04:47.549057 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49027
Sep 27 15:04:47 dist-test-slave-ml7t krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 15:04:47.794274 13489 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:47.794580 13489 heartbeater.cc:511] Master 127.0.98.62:46675 requested a full tablet report, sending...
I20260927 15:04:47.795737 13155 ts_manager.cc:194] Registered new tserver with Master: ee9f757a2fc9487d9d45af47d0eead33 (127.0.98.2:38095)
I20260927 15:04:47.796559 13155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52301
I20260927 15:04:47.802711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:04:47.824178 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.810286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.0.98.62:46675)
0927 15:04:47.814412 (+ 4126us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.814417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.814432 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:04:47.814479 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.814482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.814531 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.814612 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:47.815667 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.816492 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.817711 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.818070 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.818176 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.819999 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:47.820002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:47.820006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:47.820032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:47.821910 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:47.822397 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:47.822402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:47.822405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:47.822496 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:47.822811 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:47.822817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:47.822821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:47.823033 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:47.823155 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.823874 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.823991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4041,"thread_start_us":73,"threads_started":1}
I20260927 15:04:47.829722 13155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:04:47.834713 13155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:04:47.868278 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.852447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41665 (local address 127.0.0.1:54884)
0927 15:04:47.855444 (+ 2997us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:47.855458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:47.855547 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:47.856556 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:47.856575 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:47.856600 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:47.856746 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.856753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.858256 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.858259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.859710 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.859722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.867641 (+ 7919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.867645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.867788 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.867837 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 15:04:47.867961 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 15:04:47.868100 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2934,"thread_start_us":86,"threads_started":1}
I20260927 15:04:47.873086 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.851760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37583 (local address 127.0.0.1:56472)
0927 15:04:47.854799 (+ 3039us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:47.854850 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:47.854986 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:47.860686 (+ 5700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:47.860689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:47.860709 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:47.860812 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.860819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.862346 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.862350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.863532 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.863544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.872612 (+ 9068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.872617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.872716 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.872744 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:04:47.872831 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:04:47.872911 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2860,"thread_start_us":97,"threads_started":1}
I20260927 15:04:47.873739 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.852877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.0.98.1:41665)
0927 15:04:47.855846 (+ 2969us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.855854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.855881 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:04:47.855963 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.855969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.856191 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.856352 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:47.856893 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.858106 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.864685 (+ 6579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.865658 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.865803 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.865974 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.873424 (+ 7450us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.873558 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2853,"thread_start_us":88,"threads_started":1}
I20260927 15:04:47.877453 13288 tablet_service.cc:1467] Processing CreateTablet for tablet d76a1650b37b4e3b9ac44c7e5057eaf5 (DEFAULT_TABLE table=test-table [id=4b9c984b3f4b4ab096608cf7d97794df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:47.879628 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76a1650b37b4e3b9ac44c7e5057eaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:47.886395 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.853627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38095 (local address 127.0.0.1:41978)
0927 15:04:47.863797 (+ 10170us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:47.863813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:47.863905 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:47.866040 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:47.866044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:47.866066 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:47.866165 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.866171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.868642 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.868646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.869859 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.869872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.885859 (+ 15987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.885865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.885997 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.886030 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:04:47.886137 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:04:47.886231 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10102,"thread_start_us":101,"threads_started":1}
I20260927 15:04:47.886955 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.854069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.0.98.2:38095)
0927 15:04:47.855301 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.855309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.864018 (+ 8709us) server_negotiation.cc:407] Connection header received
0927 15:04:47.864113 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.864119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.864325 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.864476 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:47.866314 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.867462 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.870026 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.870922 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.871042 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.871196 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.886678 (+ 15482us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.886808 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":99,"threads_started":1}
I20260927 15:04:47.889181 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.851896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.0.98.3:37583)
0927 15:04:47.860022 (+ 8126us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.860029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.860054 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:04:47.860135 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.860141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.860362 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.860517 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:47.860959 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.862191 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.871337 (+ 9146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.872231 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.872375 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.872527 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.888896 (+ 16369us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.889020 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8008,"thread_start_us":96,"threads_started":1}
I20260927 15:04:47.890077 13423 tablet_service.cc:1467] Processing CreateTablet for tablet d76a1650b37b4e3b9ac44c7e5057eaf5 (DEFAULT_TABLE table=test-table [id=4b9c984b3f4b4ab096608cf7d97794df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:47.891212 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76a1650b37b4e3b9ac44c7e5057eaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:47.892393 13552 tablet_service.cc:1467] Processing CreateTablet for tablet d76a1650b37b4e3b9ac44c7e5057eaf5 (DEFAULT_TABLE table=test-table [id=4b9c984b3f4b4ab096608cf7d97794df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:47.893846 13552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76a1650b37b4e3b9ac44c7e5057eaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:47.897755 13656 tablet_bootstrap.cc:492] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33: Bootstrap starting.
I20260927 15:04:47.900899 13656 tablet_bootstrap.cc:654] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:47.901739 13656 log.cc:824] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:47.903841 13656 tablet_bootstrap.cc:492] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33: No bootstrap required, opened a new log
I20260927 15:04:47.904034 13656 ts_tablet_manager.cc:1397] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260927 15:04:47.908206 13656 raft_consensus.cc:348] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.908484 13656 raft_consensus.cc:374] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:47.908538 13656 raft_consensus.cc:729] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9f757a2fc9487d9d45af47d0eead33, State: Initialized, Role: FOLLOWER
I20260927 15:04:47.909050 13656 consensus_queue.cc:260] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.910179 13656 ts_tablet_manager.cc:1428] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 15:04:47.910526 13489 heartbeater.cc:503] Master 127.0.98.62:46675 was elected leader, sending a full tablet report...
I20260927 15:04:47.916821 13658 tablet_bootstrap.cc:492] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Bootstrap starting.
I20260927 15:04:47.917814 13655 tablet_bootstrap.cc:492] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae: Bootstrap starting.
I20260927 15:04:47.920481 13655 tablet_bootstrap.cc:654] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:47.921360 13655 log.cc:824] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:47.923198 13658 tablet_bootstrap.cc:654] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:47.923980 13658 log.cc:824] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:47.926213 13655 tablet_bootstrap.cc:492] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae: No bootstrap required, opened a new log
I20260927 15:04:47.926414 13655 ts_tablet_manager.cc:1397] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 15:04:47.931854 13655 raft_consensus.cc:348] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.932168 13655 raft_consensus.cc:374] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:47.932225 13655 raft_consensus.cc:729] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9668d4d52c684b5c9e52b95e8d11ecae, State: Initialized, Role: FOLLOWER
I20260927 15:04:47.934680 13658 tablet_bootstrap.cc:492] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: No bootstrap required, opened a new log
I20260927 15:04:47.934902 13658 ts_tablet_manager.cc:1397] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.004s
I20260927 15:04:47.939834 13658 raft_consensus.cc:348] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.940168 13658 raft_consensus.cc:374] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:47.940225 13658 raft_consensus.cc:729] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b90e7340ed84b1f9fc1d57f7572c796, State: Initialized, Role: FOLLOWER
I20260927 15:04:47.940843 13655 consensus_queue.cc:260] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.941995 13655 ts_tablet_manager.cc:1428] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260927 15:04:47.942421 13354 heartbeater.cc:503] Master 127.0.98.62:46675 was elected leader, sending a full tablet report...
I20260927 15:04:47.942581 13658 consensus_queue.cc:260] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.943493 13658 ts_tablet_manager.cc:1428] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260927 15:04:47.943799 13629 heartbeater.cc:503] Master 127.0.98.62:46675 was elected leader, sending a full tablet report...
I20260927 15:04:47.950124 13662 raft_consensus.cc:482] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:04:47.950302 13662 raft_consensus.cc:504] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.952989 13662 leader_election.cc:288] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b90e7340ed84b1f9fc1d57f7572c796 (127.0.98.3:37583), ee9f757a2fc9487d9d45af47d0eead33 (127.0.98.2:38095)
I20260927 15:04:47.960897 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.952016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52165 (local address 127.0.98.3:37583)
0927 15:04:47.952132 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.952136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.953306 (+ 1170us) server_negotiation.cc:407] Connection header received
0927 15:04:47.953519 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.953523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.953568 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.953613 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:47.954124 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.954906 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.959601 (+ 4695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.960221 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.960299 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.960364 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.960684 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.960765 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:04:47.961334 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.951663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37583 (local address 127.0.98.1:52165)
0927 15:04:47.953226 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:47.953240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:47.953374 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:47.953805 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:47.953808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:47.953825 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:47.953990 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.953997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.955128 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.955131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.956397 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.956408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.960453 (+ 4045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.960457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.960553 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.960574 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:04:47.961106 (+ 532us) client_negotiation.cc:240] Negotiation successful
0927 15:04:47.961181 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"thread_start_us":110,"threads_started":1}
I20260927 15:04:47.961987 13576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76a1650b37b4e3b9ac44c7e5057eaf5" candidate_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" is_pre_election: true
I20260927 15:04:47.962337 13576 raft_consensus.cc:2454] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9668d4d52c684b5c9e52b95e8d11ecae in term 0.
I20260927 15:04:47.962836 13242 leader_election.cc:302] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b90e7340ed84b1f9fc1d57f7572c796, 9668d4d52c684b5c9e52b95e8d11ecae; no voters:
I20260927 15:04:47.963153 13662 raft_consensus.cc:2781] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:04:47.963234 13662 raft_consensus.cc:482] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:04:47.963280 13662 raft_consensus.cc:3037] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:47.964601 13662 raft_consensus.cc:504] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.965263 13576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76a1650b37b4e3b9ac44c7e5057eaf5" candidate_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b90e7340ed84b1f9fc1d57f7572c796"
I20260927 15:04:47.965389 13576 raft_consensus.cc:3037] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:47.966714 13576 raft_consensus.cc:2454] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9668d4d52c684b5c9e52b95e8d11ecae in term 1.
I20260927 15:04:47.967020 13242 leader_election.cc:302] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b90e7340ed84b1f9fc1d57f7572c796, 9668d4d52c684b5c9e52b95e8d11ecae; no voters:
I20260927 15:04:47.967430 13662 leader_election.cc:288] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [CANDIDATE]: Term 1 election: Requested vote from peers 1b90e7340ed84b1f9fc1d57f7572c796 (127.0.98.3:37583), ee9f757a2fc9487d9d45af47d0eead33 (127.0.98.2:38095)
I20260927 15:04:47.967528 13662 raft_consensus.cc:2781] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:04:47.967600 13662 raft_consensus.cc:686] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [term 1 LEADER]: Becoming Leader. State: Replica: 9668d4d52c684b5c9e52b95e8d11ecae, State: Running, Role: LEADER
I20260927 15:04:47.967978 13662 consensus_queue.cc:237] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } }
I20260927 15:04:47.971441 13155 catalog_manager.cc:5495] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae reported cstate change: term changed from 0 to 1, leader changed from <none> to 9668d4d52c684b5c9e52b95e8d11ecae (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 } health_report { overall_health: UNKNOWN } } }
I20260927 15:04:47.977180 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.952851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34139 (local address 127.0.98.2:38095)
0927 15:04:47.952956 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:47.952960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:47.956737 (+ 3777us) server_negotiation.cc:407] Connection header received
0927 15:04:47.956911 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:47.956914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:47.956964 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:47.957012 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:47.957386 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.958150 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.975754 (+ 17604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.976441 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.976532 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.976599 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:47.976957 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 15:04:47.977040 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:04:47.977592 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:47.952496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38095 (local address 127.0.98.1:34139)
0927 15:04:47.956657 (+ 4161us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:47.956671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:47.956801 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:47.957152 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:47.957155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:47.957170 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:47.957269 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.957273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.958287 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.958290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.959456 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:47.959466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:47.976704 (+ 17238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:47.976708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:47.976821 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:47.976843 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:04:47.977372 (+ 529us) client_negotiation.cc:240] Negotiation successful
0927 15:04:47.977455 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4081,"thread_start_us":96,"threads_started":1}
I20260927 15:04:47.978287 13442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76a1650b37b4e3b9ac44c7e5057eaf5" candidate_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9f757a2fc9487d9d45af47d0eead33"
I20260927 15:04:47.978245 13443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76a1650b37b4e3b9ac44c7e5057eaf5" candidate_uuid: "9668d4d52c684b5c9e52b95e8d11ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9f757a2fc9487d9d45af47d0eead33" is_pre_election: true
I20260927 15:04:47.978557 13443 raft_consensus.cc:2454] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9668d4d52c684b5c9e52b95e8d11ecae in term 0.
W20260927 15:04:48.002223 13355 tablet.cc:2369] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:04:48.015515 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:48.009950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.0.98.1:41665)
0927 15:04:48.010072 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:48.010076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:48.010436 (+ 360us) server_negotiation.cc:407] Connection header received
0927 15:04:48.010642 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:48.010646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:48.010720 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:48.010824 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:04:48.011211 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:48.011966 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:48.013211 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:48.013523 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:48.013617 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:48.014097 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:04:48.014713 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:04:48.014835 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:48.015286 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 15:04:48.015354 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 15:04:48.019477 13490 tablet.cc:2369] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:04:48.024065 13442 raft_consensus.cc:3037] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:48.025616 13442 raft_consensus.cc:1261] T d76a1650b37b4e3b9ac44c7e5057eaf5 P ee9f757a2fc9487d9d45af47d0eead33 [term 1 FOLLOWER]: Refusing update from remote peer 9668d4d52c684b5c9e52b95e8d11ecae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 15:04:48.025929 13630 tablet.cc:2369] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:04:48.026959 13662 consensus_queue.cc:1035] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9f757a2fc9487d9d45af47d0eead33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:48.034976 13576 raft_consensus.cc:1261] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 1 FOLLOWER]: Refusing update from remote peer 9668d4d52c684b5c9e52b95e8d11ecae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:04:48.035694 13662 consensus_queue.cc:1035] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 9668d4d52c684b5c9e52b95e8d11ecae [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b90e7340ed84b1f9fc1d57f7572c796" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:48.052222 13669 mvcc.cc:205] Tried to move back new op lower bound from 7333976014938124288 to 7333976014720131072. Current Snapshot: MvccSnapshot[applied={T|T < 7333976014938124288}]
I20260927 15:04:48.054970 13672 mvcc.cc:205] Tried to move back new op lower bound from 7333976014938124288 to 7333976014720131072. Current Snapshot: MvccSnapshot[applied={T|T < 7333976014938124288}]
I20260927 15:04:48.066789 13670 mvcc.cc:205] Tried to move back new op lower bound from 7333976014938124288 to 7333976014720131072. Current Snapshot: MvccSnapshot[applied={T|T < 7333976014938124288}]
I20260927 15:04:48.076830 13155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:04:48.077031 13155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:04:48.079924 13155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddc2d8ef96c1410f81084c927eb99b8f: Sending DeleteTablet for 3 replicas of tablet d76a1650b37b4e3b9ac44c7e5057eaf5
I20260927 15:04:48.081365 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet d76a1650b37b4e3b9ac44c7e5057eaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41978
I20260927 15:04:48.081377 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet d76a1650b37b4e3b9ac44c7e5057eaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54884
I20260927 15:04:48.082302 13552 tablet_service.cc:1514] Processing DeleteTablet for tablet d76a1650b37b4e3b9ac44c7e5057eaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56472
I20260927 15:04:48.083047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 13194
W20260927 15:04:48.091427 13142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41665 (error 108)
I20260927 15:04:48.091881 13680 tablet_replica.cc:330] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: stopping tablet replica
I20260927 15:04:48.092419 13680 raft_consensus.cc:2229] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:04:48.092965 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:48.092641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41665 (local address 127.0.0.1:54912)
0927 15:04:48.092819 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:48.092880 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 15:04:48.092963 13680 raft_consensus.cc:2258] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 15:04:48.093134 13142 catalog_manager.cc:4584] TS 9668d4d52c684b5c9e52b95e8d11ecae (127.0.98.1:41665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d76a1650b37b4e3b9ac44c7e5057eaf5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41665: connect: Connection refused (error 111)
I20260927 15:04:48.093448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 13357
I20260927 15:04:48.097038 13680 ts_tablet_manager.cc:1905] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:48.099972 13680 ts_tablet_manager.cc:1918] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:48.100077 13680 log.cc:1196] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/d76a1650b37b4e3b9ac44c7e5057eaf5
I20260927 15:04:48.100402 13680 ts_tablet_manager.cc:1939] T d76a1650b37b4e3b9ac44c7e5057eaf5 P 1b90e7340ed84b1f9fc1d57f7572c796: Deleting consensus metadata
I20260927 15:04:48.101821 13141 catalog_manager.cc:4841] TS 1b90e7340ed84b1f9fc1d57f7572c796 (127.0.98.3:37583): tablet d76a1650b37b4e3b9ac44c7e5057eaf5 (table test-table [id=4b9c984b3f4b4ab096608cf7d97794df]) successfully deleted
I20260927 15:04:48.108071 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:48.107799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38095 (local address 127.0.0.1:41988)
0927 15:04:48.107916 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:48.107972 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 15:04:48.108527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 13492
I20260927 15:04:48.116030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 13123
I20260927 15:04:48.124795 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:04:48.469731 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:04:48.469933 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:04:48.814779 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:04:48.815 UTC [12190] LOG: received smart shutdown request
2026-09-27 15:04:48.818 UTC [12190] LOG: background worker "logical replication launcher" (PID 12199) exited with exit code 1
2026-09-27 15:04:48.819 UTC [12194] LOG: shutting down
2026-09-27 15:04:48.989 UTC [12190] LOG: database system is shut down
2026-09-27T15:04:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49256 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:04:49 dist-test-slave-ml7t krb5kdc[13691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:04:49 dist-test-slave-ml7t krb5kdc[13691](info): set up 2 sockets
Sep 27 15:04:49 dist-test-slave-ml7t krb5kdc[13691](info): commencing operation
krb5kdc: starting...
W20260927 15:04:50.153914 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:04:50.276495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41935,127.0.98.61:44233,127.0.98.60:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:50.383190 13704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:50.383438 13704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:50.383478 13704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:50.387476 13704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:50.387532 13704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:50.387552 13704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:50.387569 13704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:50.387585 13704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:50.392786 13704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41935,127.0.98.61:44233,127.0.98.60:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:50.393750 13704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:50.394776 13704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:50.402796 13712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:50.402866 13710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:50.402781 13709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:50.403100 13704 server_base.cc:1034] running on GCE node
I20260927 15:04:50.403625 13704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:50.404700 13704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:50.405870 13704 hybrid_clock.cc:648] HybridClock initialized: now 1790521490405840 us; error 45 us; skew 500 ppm
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:50.408656 13704 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:04:50.409487 13704 webserver.cc:460] Webserver started at http://127.0.98.62:46739/ using document root <none> and password file <none>
I20260927 15:04:50.410089 13704 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:50.410183 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:50.410413 13704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:50.412540 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "c365cb6561fe49c2822ccb4df8ad8ade"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "64482ed241c30af77423009453f671bb"
server_key_iv: "91f0249ac2a6092f8aa859733d116017"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.413147 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "c365cb6561fe49c2822ccb4df8ad8ade"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "64482ed241c30af77423009453f671bb"
server_key_iv: "91f0249ac2a6092f8aa859733d116017"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.416965 13704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 15:04:50.419450 13719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:50.420668 13704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:04:50.420807 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "c365cb6561fe49c2822ccb4df8ad8ade"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "64482ed241c30af77423009453f671bb"
server_key_iv: "91f0249ac2a6092f8aa859733d116017"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.420915 13704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:50.462218 13704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:50.465260 13704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:50.465464 13704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:50.472397 13704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41935
I20260927 15:04:50.472460 13771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41935 every 8 connection(s)
I20260927 15:04:50.473376 13704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:04:50.476051 13772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:50.480108 13772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:04:50.483357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 13704
I20260927 15:04:50.483479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:04:50.483775 392 external_mini_cluster.cc:1428] Setting key 4e6204f86be920dd5e092abe79dc5b91
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 15:04:50.496950 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.482704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.0.98.62:41935)
0927 15:04:50.483710 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.483719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.483755 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:04:50.484872 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.484877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.485061 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.485242 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.486194 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.487248 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.488859 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.489230 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.489343 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.492252 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.492258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.492268 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.492292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.493905 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.494386 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.494392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.494394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.494496 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.494834 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.494840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.494841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.495381 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.495524 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.495901 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.496134 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":894,"thread_start_us":151,"threads_started":1}
I20260927 15:04:50.496956 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.482704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41935 (local address 127.0.0.1:57658)
0927 15:04:50.483287 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.483350 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.484544 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.485270 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.485277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.485658 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.486049 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.486062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.487377 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.487383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.488718 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.488732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.489390 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.489398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.489560 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.490261 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.490283 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.492095 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.494023 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.494030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.494040 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.494276 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.494588 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.494591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.494593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.494731 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.495509 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.495514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.495642 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.495797 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.496053 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":60,"spinlock_wait_cycles":53120,"thread_start_us":168,"threads_started":1}
I20260927 15:04:50.499689 13772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:04:50.500423 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.500179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60636)
0927 15:04:50.500306 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.500370 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 15:04:50.500703 13720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
I20260927 15:04:50.501529 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.501332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60638)
0927 15:04:50.501440 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.501481 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:04:50.501713 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44233: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 15:04:50.511022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44233
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41935,127.0.98.61:44233,127.0.98.60:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:50.550992 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } attempt: 1
I20260927 15:04:50.551985 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.551651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60648)
0927 15:04:50.551837 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.551908 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 15:04:50.552860 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.552640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60664)
0927 15:04:50.552761 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.552806 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:04:50.553043 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44233: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
I20260927 15:04:50.621212 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } attempt: 2
I20260927 15:04:50.622116 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.621822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60678)
0927 15:04:50.621973 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.622040 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:04:50.622804 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.622619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60686)
0927 15:04:50.622711 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.622752 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:04:50.622957 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44233: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44233: connect: Connection refused (error 111)
W20260927 15:04:50.622833 13779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:50.623142 13779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:50.623188 13779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:50.627328 13779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:50.627381 13779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:50.627404 13779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:50.627422 13779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:50.627440 13779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:50.632529 13779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41935,127.0.98.61:44233,127.0.98.60:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:50.633448 13779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:50.634701 13779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:50.643280 13784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:50.643342 13785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:50.643455 13779 server_base.cc:1034] running on GCE node
W20260927 15:04:50.643280 13787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:50.644091 13779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:50.645147 13779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:50.646347 13779 hybrid_clock.cc:648] HybridClock initialized: now 1790521490646328 us; error 31 us; skew 500 ppm
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:50.649310 13779 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 15:04:50.650239 13779 webserver.cc:460] Webserver started at http://127.0.98.61:42637/ using document root <none> and password file <none>
I20260927 15:04:50.650832 13779 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:50.650943 13779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:50.651247 13779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:50.653430 13779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data/instance:
uuid: "061b00fb36674ca98cdb8a5dd194d383"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "e45da85d84bd4f39e67a45e51975be8c"
server_key_iv: "31b6f3f025258ec095120145ec566c59"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.654057 13779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/wal/instance:
uuid: "061b00fb36674ca98cdb8a5dd194d383"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "e45da85d84bd4f39e67a45e51975be8c"
server_key_iv: "31b6f3f025258ec095120145ec566c59"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.657867 13779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:04:50.660411 13794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:50.661623 13779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:04:50.661801 13779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/wal
uuid: "061b00fb36674ca98cdb8a5dd194d383"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "e45da85d84bd4f39e67a45e51975be8c"
server_key_iv: "31b6f3f025258ec095120145ec566c59"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.661923 13779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:50.679190 13779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:50.682489 13779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:50.682694 13779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:50.689831 13779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44233
I20260927 15:04:50.689869 13846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44233 every 8 connection(s)
I20260927 15:04:50.690820 13779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
I20260927 15:04:50.693498 13847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:50.697280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 13779
I20260927 15:04:50.697399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-1/wal/instance
I20260927 15:04:50.697228 13847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:04:50.697679 392 external_mini_cluster.cc:1428] Setting key ce778277ae976513cc506fcf335f94a6
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:50.713208 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.699473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.98.62:41935)
0927 15:04:50.699659 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.699665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.700481 (+ 816us) server_negotiation.cc:407] Connection header received
0927 15:04:50.701427 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.701433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.701501 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.701627 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.702791 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.703700 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.704859 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.705282 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.705400 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.709180 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.709184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.709187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.709222 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.710783 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.711352 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.711358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.711363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.711466 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.711819 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.711825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.711829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.712104 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.712222 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.712818 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.712972 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:04:50.713984 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.699859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41935 (local address 127.0.0.1:57668)
0927 15:04:50.700324 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.700375 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.701196 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.701817 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.701831 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.702161 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.702635 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.702648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.703842 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.703845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.704726 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.704736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.705437 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.705439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.705577 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.706310 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.706327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.708998 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.710918 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.710923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.710937 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.711232 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.711563 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.711565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.711566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.711704 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.712212 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.712217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.712549 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.712712 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.712990 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260927 15:04:50.714121 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } attempt: 3
I20260927 15:04:50.715759 13847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 15:04:50.725576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36921
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41935,127.0.98.61:44233,127.0.98.60:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:50.725967 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.714551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60688)
0927 15:04:50.714897 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.714911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.714992 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.715532 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.715537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.715727 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.716335 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.716345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.717572 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.717577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.718242 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.718251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.718883 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.718887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.718994 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.719488 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.719504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.720758 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.724135 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.724138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.724140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.724491 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.724837 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.724840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.724842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.724929 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.725414 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.725417 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.725520 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.725631 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.725762 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260927 15:04:50.726122 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.714670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.0.98.61:44233)
0927 15:04:50.715097 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.715107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.715147 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:04:50.715188 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.715194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.715386 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.715515 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.716484 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.717454 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.718361 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.718773 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.718886 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.720891 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.720899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.720910 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.720930 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.724011 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.724639 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.724646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.724651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.724749 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.725002 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.725007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.725009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.725280 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.725419 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.725715 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.725900 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":342,"thread_start_us":154,"threads_started":1}
I20260927 15:04:50.726292 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.716232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60692)
0927 15:04:50.716387 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.716401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.716476 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.716762 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.716764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.716998 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.717074 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.717079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.717917 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.717923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.718465 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.718473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.719143 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.719146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.719230 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.719651 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.719662 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.721345 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.724099 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.724104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.724108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.724412 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.724751 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.724754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.724756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.724837 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.725461 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.725463 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.725549 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.725653 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.725819 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 15:04:50.726423 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.716266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.0.98.61:44233)
0927 15:04:50.716588 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.716594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.716604 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:04:50.716627 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.716629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.716657 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.716749 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.717181 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.717807 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.718638 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.719032 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.719145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.721498 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.721502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.721504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.721537 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.723954 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.724532 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.724541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.724546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.724639 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.724923 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.724927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.724930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.725293 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.725421 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.726043 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.726139 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260927 15:04:50.727995 13847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:04:50.728116 13772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:04:50.728736 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.728486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34704)
0927 15:04:50.728643 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.728683 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 15:04:50.728767 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.728486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34694)
0927 15:04:50.728608 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.728690 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 15:04:50.729013 13795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
I20260927 15:04:50.729458 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.729179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34710)
0927 15:04:50.729360 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.729405 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260927 15:04:50.729614 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
I20260927 15:04:50.729964 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.729704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34712)
0927 15:04:50.729857 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.729912 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 15:04:50.730129 13847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
I20260927 15:04:50.760789 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } attempt: 1
I20260927 15:04:50.761632 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.761335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34726)
0927 15:04:50.761484 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.761555 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:04:50.762405 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.762183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34736)
0927 15:04:50.762302 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.762352 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:04:50.762537 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
I20260927 15:04:50.779367 13847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } attempt: 1
I20260927 15:04:50.780211 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.779931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34744)
0927 15:04:50.780075 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.780137 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:04:50.780925 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.780710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34754)
0927 15:04:50.780817 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.780870 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:04:50.781050 13847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
I20260927 15:04:50.837646 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } attempt: 2
W20260927 15:04:50.838233 13854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:50.838523 13854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:50.838598 13854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:04:50.838608 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.838312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34766)
0927 15:04:50.838451 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.838513 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 15:04:50.839367 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.839140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34780)
0927 15:04:50.839261 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.839314 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:04:50.839525 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
W20260927 15:04:50.843103 13854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:50.843161 13854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:50.843182 13854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:50.843226 13854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:50.843261 13854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:50.848604 13854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41935,127.0.98.61:44233,127.0.98.60:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:50.849201 13847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } attempt: 2
I20260927 15:04:50.849731 13854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:50.850003 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.849674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34792)
0927 15:04:50.849841 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.849906 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:04:50.850709 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.850513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34798)
0927 15:04:50.850605 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.850650 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 15:04:50.850839 13847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36921: connect: Connection refused (error 111)
I20260927 15:04:50.851040 13854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:50.859447 13860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:50.859452 13863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:50.859470 13861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:50.859722 13854 server_base.cc:1034] running on GCE node
I20260927 15:04:50.860267 13854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:50.861479 13854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:50.862676 13854 hybrid_clock.cc:648] HybridClock initialized: now 1790521490862647 us; error 40 us; skew 500 ppm
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:50.865406 13854 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 15:04:50.866299 13854 webserver.cc:460] Webserver started at http://127.0.98.60:40093/ using document root <none> and password file <none>
I20260927 15:04:50.866799 13854 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:50.866855 13854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:50.867018 13854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:50.868914 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data/instance:
uuid: "98aa32d4a4254d28984d31eb6dcf2074"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "7d154c2789313069dddd80e0ca743e98"
server_key_iv: "95e205c704086548a4ac5871602bd1ba"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.869388 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/wal/instance:
uuid: "98aa32d4a4254d28984d31eb6dcf2074"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "7d154c2789313069dddd80e0ca743e98"
server_key_iv: "95e205c704086548a4ac5871602bd1ba"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.872942 13854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:04:50.875260 13870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:50.876449 13854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:04:50.876583 13854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/wal
uuid: "98aa32d4a4254d28984d31eb6dcf2074"
format_stamp: "Formatted at 2026-09-27 15:04:50 on dist-test-slave-ml7t"
server_key: "7d154c2789313069dddd80e0ca743e98"
server_key_iv: "95e205c704086548a4ac5871602bd1ba"
server_key_version: "encryptionkey@0"
I20260927 15:04:50.876729 13854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:50.894165 13854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:50.897394 13854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:50.897600 13854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:50.904532 13854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36921
I20260927 15:04:50.904556 13922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36921 every 8 connection(s)
I20260927 15:04:50.905437 13854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
I20260927 15:04:50.908128 13923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:50.911939 13923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:04:50.912818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 13854
I20260927 15:04:50.912911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521328452215-392-0/minicluster-data/master-2/wal/instance
I20260927 15:04:50.913218 392 external_mini_cluster.cc:1428] Setting key 573f660da31b1a43f7f7aacae05e14b2
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:04:50.926229 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.913933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.0.98.62:41935)
0927 15:04:50.914068 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.914073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.915459 (+ 1386us) server_negotiation.cc:407] Connection header received
0927 15:04:50.915761 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.915765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.915811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.915900 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.917002 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.917796 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.919116 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.919459 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.919568 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.922336 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.922340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.922342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.922365 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.923991 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.924614 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.924618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.924620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.924710 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.925085 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.925088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.925089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.925374 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.925476 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.925925 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.926051 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:04:50.927021 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.914393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41935 (local address 127.0.0.1:57680)
0927 15:04:50.915312 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.915349 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.915574 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.915993 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.915998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.916482 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.916856 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.916875 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.917896 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.917898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.918996 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.919006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.919603 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.919606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.919799 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.920438 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.920466 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.922143 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.924119 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.924129 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.924139 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.924494 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.924819 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.924822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.924824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.924975 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.925479 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.925483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.925598 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.925844 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.926151 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":27,"thread_start_us":150,"threads_started":1}
I20260927 15:04:50.927028 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.914393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.0.98.60:36921)
0927 15:04:50.914825 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.914838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.914869 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:04:50.915564 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.915580 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.915815 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.915983 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.916857 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.918074 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.919184 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.919770 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.919847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.922744 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.922749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.922758 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.922777 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.924305 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.924752 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.924759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.924762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.924875 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.925158 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.925161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.925163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.925585 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.925921 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.926074 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.926197 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":308,"spinlock_wait_cycles":56960,"thread_start_us":149,"threads_started":1}
I20260927 15:04:50.928294 13923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:50.937479 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.928767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44233 (local address 127.0.0.1:60694)
0927 15:04:50.928989 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.929000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.929089 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.929470 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.929473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.929670 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.929811 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.929820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.930713 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.930716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.931453 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.931462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.931950 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.931953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.932024 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.932513 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.932533 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.934026 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.935863 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.935867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.935869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.936061 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.936384 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.936386 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.936388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.936466 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.936934 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.936936 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.937038 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.937148 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.937296 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260927 15:04:50.937752 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.929330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.0.98.62:41935)
0927 15:04:50.929437 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.929443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.929460 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:04:50.929561 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.929563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.929595 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.929656 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.930326 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.930935 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.931854 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.932140 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.932272 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.934623 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.934631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.934636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.934664 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.936150 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.936557 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.936562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.936567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.936651 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.936932 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.936935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.936937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.937148 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.937400 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.937477 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.937606 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:04:50.937537 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.928885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.0.98.61:44233)
0927 15:04:50.929025 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.929032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.929049 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:04:50.929211 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.929214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.929273 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.929353 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.929942 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.930600 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.931569 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.931839 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.931944 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.934200 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.934203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.934206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.934247 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.935736 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.936166 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.936170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.936172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.936273 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.936579 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.936582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.936584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.936835 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.936931 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.937244 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.937371 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:04:50.938551 13923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:04:50.938700 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } attempt: 3
I20260927 15:04:50.941993 13847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } attempt: 3
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:04:50.948163 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.939118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34812)
0927 15:04:50.939288 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.939300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.939363 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.939600 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.939603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.939768 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.939870 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.939879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.940917 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.940920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.941702 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.941711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.942335 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.942338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.942412 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.942909 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.942920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.944315 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.946388 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.946392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.946396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.946684 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.946968 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.946973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.946976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.947054 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.947536 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.947540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.947640 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.947833 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.947963 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 15:04:50.948669 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.939246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.0.98.60:36921)
0927 15:04:50.939358 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.939362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.939376 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:04:50.939440 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.939446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.939495 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.939573 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.939977 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.940785 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.941827 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.942129 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.942231 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.944468 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.944474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.944481 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.944511 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.946294 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.946788 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.946794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.946796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.946875 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.947149 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.947154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.947157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.947398 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.947500 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.948408 (+ 908us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.948519 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 15:04:50 dist-test-slave-ml7t krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:04:50.950304 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.939098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34814)
0927 15:04:50.939214 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.939229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.939307 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.939961 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.939966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.940133 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.940225 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.940232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.941734 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.941737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.942542 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.942553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.943185 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.943189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.943301 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.943841 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.943856 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.945870 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.948402 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.948405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.948407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.948650 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.949215 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.949218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.949220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.949344 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.949811 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.949814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.949911 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.949998 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.950135 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 15:04:50.950805 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.939327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.0.98.60:36921)
0927 15:04:50.939691 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.939699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.939713 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:04:50.939780 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.939783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.939817 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.940333 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.940409 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.941617 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.942670 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.943030 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.943136 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.946021 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.946027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.946030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.946055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.948271 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.949009 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.949013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.949015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.949104 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.949460 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.949463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.949465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.949674 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.950475 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.950545 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.950646 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20260927 15:04:50.952070 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.939962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.0.98.61:44233)
0927 15:04:50.940067 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.940070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.940080 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:04:50.940106 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.940108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.940138 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.940237 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.940809 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.941281 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.942239 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.942833 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.942905 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.946602 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.946607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.946611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.946640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.949700 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.950231 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.950235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.950237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.950313 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.950645 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.950652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.950654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.951180 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.951391 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.951819 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.951932 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:04:50.952335 13923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074: Bootstrap starting.
I20260927 15:04:50.953828 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.942363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36921 (local address 127.0.0.1:34830)
0927 15:04:50.942619 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:50.942633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:50.942701 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:50.944722 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:50.944726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:50.944898 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:50.945004 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.945014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.946096 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.946099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.946962 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:50.946972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.947626 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.947631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:50.947725 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.948112 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:50.948129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:50.950107 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:50.952329 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.952332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.952334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.952549 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.952845 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:50.952851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:50.952853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:50.952938 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:50.953342 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:50.953348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:50.953441 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 15:04:50.953541 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:04:50.953657 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260927 15:04:50.954303 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.942598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.98.60:36921)
0927 15:04:50.943545 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.943554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.943568 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:04:50.944532 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.944537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.944580 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.944680 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.945114 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.945957 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.947106 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.947485 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.947926 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.950267 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.950274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.950279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.950302 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.952193 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.952682 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.952685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.952687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.952770 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.953042 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.953045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.953047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.953247 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.953337 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.954009 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.954122 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":114,"threads_started":1}
I20260927 15:04:50.955196 13923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:50.955686 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:04:50.956063 13923 log.cc:824] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:50.957314 13847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: Bootstrap starting.
I20260927 15:04:50.960168 13847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:50.960135 13772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: Bootstrap starting.
I20260927 15:04:50.961112 13847 log.cc:824] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:50.962812 13772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:50.963660 13847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: No bootstrap required, opened a new log
I20260927 15:04:50.963753 13772 log.cc:824] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:50.965317 13923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074: No bootstrap required, opened a new log
I20260927 15:04:50.968286 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.958544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.0.98.61:44233)
0927 15:04:50.958638 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.958641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.959594 (+ 953us) server_negotiation.cc:407] Connection header received
0927 15:04:50.959777 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.959780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.959825 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.959873 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.960814 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.961508 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.962590 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.962932 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.963013 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.964848 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.964851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.964854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.964878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.966463 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.966952 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.966956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.966958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.967038 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.967329 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.967332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.967334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.967538 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.967641 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.968020 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.968131 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:04:50.968681 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.957821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.0.98.62:41935)
0927 15:04:50.957938 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.957943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.958426 (+ 483us) server_negotiation.cc:407] Connection header received
0927 15:04:50.958619 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.958624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.958675 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.958740 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.959696 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.960449 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.961884 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.962253 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.962364 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.964410 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.964413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.964416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.964441 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.966647 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.967184 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.967188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.967189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.967286 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.967575 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.967578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.967580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.967782 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.967899 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.968426 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.968534 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:04:50.968581 13847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:50.968808 13847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:50.968859 13847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061b00fb36674ca98cdb8a5dd194d383, State: Initialized, Role: FOLLOWER
I20260927 15:04:50.969538 13847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:50.970366 13923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:50.970548 13923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:50.970582 13923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98aa32d4a4254d28984d31eb6dcf2074, State: Initialized, Role: FOLLOWER
I20260927 15:04:50.970924 13847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:50.971047 13923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:50.971517 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:50.971735 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:04:50.972242 13923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:50.973469 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:50.974658 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:04:50.975663 13772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: No bootstrap required, opened a new log
W20260927 15:04:50.976285 13957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:04:50.976406 13957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:04:50.974817 13779 master_runner.cc:186] Error getting master registration for 127.0.98.62:41935: OK, instance_id { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" instance_seqno: 1790521490470557 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:04:50.978127 13779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:04:50.980666 13964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:04:50.980753 13964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:04:50.979197 13854 master_runner.cc:186] Error getting master registration for 127.0.98.62:41935: OK, instance_id { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" instance_seqno: 1790521490470557 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:04:50.982553 13854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:04:50.984694 13772 raft_consensus.cc:348] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:50.984956 13772 raft_consensus.cc:374] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:50.985018 13772 raft_consensus.cc:729] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c365cb6561fe49c2822ccb4df8ad8ade, State: Initialized, Role: FOLLOWER
I20260927 15:04:50.985528 13772 consensus_queue.cc:260] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:50.986056 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:50.986270 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:04:50.986971 13772 sys_catalog.cc:564] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:50.987108 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:50.959119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.98.60:36921)
0927 15:04:50.961874 (+ 2755us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:50.961879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:50.963911 (+ 2032us) server_negotiation.cc:407] Connection header received
0927 15:04:50.963961 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:50.963964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:50.964018 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:50.964102 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:50.977148 (+ 13046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.978794 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.981036 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:50.981414 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:50.981524 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:50.983236 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:50.983243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:50.983247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:50.983277 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:50.984888 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.985441 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.985444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.985445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.985513 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:50.985829 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:50.985833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:50.985835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:50.986062 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:50.986799 (+ 737us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:50.986863 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 15:04:50.986959 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2678}
W20260927 15:04:50.992034 13977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:04:50.992177 13977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:04:50.996869 13704 master_runner.cc:263] No leader master found from master c365cb6561fe49c2822ccb4df8ad8ade
I20260927 15:04:50.996946 13704 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:04:51.149006 13876 tablet.cc:2369] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:04:51.184207 13800 tablet.cc:2369] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:04:51.217278 13725 tablet.cc:2369] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:04:51.233819 13941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:04:51.234045 13941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:51.235081 13941 leader_election.cc:288] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c365cb6561fe49c2822ccb4df8ad8ade (127.0.98.62:41935), 061b00fb36674ca98cdb8a5dd194d383 (127.0.98.61:44233)
I20260927 15:04:51.235723 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98aa32d4a4254d28984d31eb6dcf2074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" is_pre_election: true
I20260927 15:04:51.235723 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98aa32d4a4254d28984d31eb6dcf2074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061b00fb36674ca98cdb8a5dd194d383" is_pre_election: true
I20260927 15:04:51.236056 13746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98aa32d4a4254d28984d31eb6dcf2074 in term 0.
I20260927 15:04:51.236056 13821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98aa32d4a4254d28984d31eb6dcf2074 in term 0.
I20260927 15:04:51.236527 13871 leader_election.cc:302] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 061b00fb36674ca98cdb8a5dd194d383, 98aa32d4a4254d28984d31eb6dcf2074, c365cb6561fe49c2822ccb4df8ad8ade; no voters:
I20260927 15:04:51.236853 13941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:04:51.236960 13941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:04:51.237021 13941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:51.238170 13941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:51.238492 13941 leader_election.cc:288] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [CANDIDATE]: Term 1 election: Requested vote from peers c365cb6561fe49c2822ccb4df8ad8ade (127.0.98.62:41935), 061b00fb36674ca98cdb8a5dd194d383 (127.0.98.61:44233)
I20260927 15:04:51.238816 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98aa32d4a4254d28984d31eb6dcf2074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c365cb6561fe49c2822ccb4df8ad8ade"
I20260927 15:04:51.238816 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98aa32d4a4254d28984d31eb6dcf2074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061b00fb36674ca98cdb8a5dd194d383"
I20260927 15:04:51.238941 13746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:51.238941 13821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:51.240187 13821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98aa32d4a4254d28984d31eb6dcf2074 in term 1.
I20260927 15:04:51.240187 13746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98aa32d4a4254d28984d31eb6dcf2074 in term 1.
I20260927 15:04:51.240458 13871 leader_election.cc:302] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061b00fb36674ca98cdb8a5dd194d383, 98aa32d4a4254d28984d31eb6dcf2074; no voters:
I20260927 15:04:51.240684 13941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:04:51.240890 13941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [term 1 LEADER]: Becoming Leader. State: Replica: 98aa32d4a4254d28984d31eb6dcf2074, State: Running, Role: LEADER
I20260927 15:04:51.241201 13941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } }
I20260927 15:04:51.243248 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98aa32d4a4254d28984d31eb6dcf2074. Latest consensus state: current_term: 1 leader_uuid: "98aa32d4a4254d28984d31eb6dcf2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:51.243369 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:51.243709 13981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:04:51.244521 13981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:04:51.248838 13746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [term 1 FOLLOWER]: Refusing update from remote peer 98aa32d4a4254d28984d31eb6dcf2074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:04:51.248886 13821 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [term 1 FOLLOWER]: Refusing update from remote peer 98aa32d4a4254d28984d31eb6dcf2074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:04:51.249509 13979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:51.249645 13941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:51.253337 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98aa32d4a4254d28984d31eb6dcf2074. Latest consensus state: current_term: 1 leader_uuid: "98aa32d4a4254d28984d31eb6dcf2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:51.253469 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:04:51.253523 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98aa32d4a4254d28984d31eb6dcf2074. Latest consensus state: current_term: 1 leader_uuid: "98aa32d4a4254d28984d31eb6dcf2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:51.253696 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:04:51.256294 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98aa32d4a4254d28984d31eb6dcf2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:51.256431 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:51.256672 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98aa32d4a4254d28984d31eb6dcf2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:51.256791 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:51.257273 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98aa32d4a4254d28984d31eb6dcf2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:51.257395 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:04:51.257510 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98aa32d4a4254d28984d31eb6dcf2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c365cb6561fe49c2822ccb4df8ad8ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } peers { permanent_uuid: "061b00fb36674ca98cdb8a5dd194d383" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44233 } } peers { permanent_uuid: "98aa32d4a4254d28984d31eb6dcf2074" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36921 } } }
I20260927 15:04:51.257624 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:04:51.257887 13982 mvcc.cc:205] Tried to move back new op lower bound from 7333976028148940800 to 7333976028126470144. Current Snapshot: MvccSnapshot[applied={T|T < 7333976028148940800}]
I20260927 15:04:51.260265 13981 catalog_manager.cc:1347] Generated new cluster ID: 4c11671e0a084c62b87b6c5e903e204f
I20260927 15:04:51.260319 13981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:04:51.304469 13981 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:04:51.305666 13981 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:04:51.312712 13981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98aa32d4a4254d28984d31eb6dcf2074: Generated new TSK 0
I20260927 15:04:51.313602 13981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:04:51.977845 13957 catalog_manager.cc:1259] Loaded cluster ID: 4c11671e0a084c62b87b6c5e903e204f
I20260927 15:04:51.977953 13957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: loading cluster ID for follower catalog manager: success
I20260927 15:04:51.979655 13957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: acquiring CA information for follower catalog manager: success
I20260927 15:04:51.980983 13957 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 061b00fb36674ca98cdb8a5dd194d383: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:04:51.993554 13977 catalog_manager.cc:1259] Loaded cluster ID: 4c11671e0a084c62b87b6c5e903e204f
I20260927 15:04:51.993642 13977 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: loading cluster ID for follower catalog manager: success
I20260927 15:04:51.995395 13977 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: acquiring CA information for follower catalog manager: success
I20260927 15:04:51.996558 13977 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c365cb6561fe49c2822ccb4df8ad8ade: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:04:52.389182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 13704
I20260927 15:04:52.395453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 13779
I20260927 15:04:52.401031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 13854
2026-09-27T15:04:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3288 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:04:52 dist-test-slave-ml7t krb5kdc[13998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:04:52 dist-test-slave-ml7t krb5kdc[13998](info): set up 2 sockets
Sep 27 15:04:52 dist-test-slave-ml7t krb5kdc[13998](info): commencing operation
krb5kdc: starting...
W20260927 15:04:53.466092 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:04:53 dist-test-slave-ml7t krb5kdc[13998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:04:53.592869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33597
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:53.702307 14011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:53.702625 14011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:53.702704 14011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:53.706825 14011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:53.706925 14011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:53.706964 14011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:53.707023 14011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:53.707060 14011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:53.712406 14011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:53.713516 14011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:53.714618 14011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:53.723392 14016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:53.723436 14017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:53.723400 14019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:53.724013 14011 server_base.cc:1034] running on GCE node
I20260927 15:04:53.724509 14011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:53.725553 14011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:53.726704 14011 hybrid_clock.cc:648] HybridClock initialized: now 1790521493726687 us; error 29 us; skew 500 ppm
Sep 27 15:04:53 dist-test-slave-ml7t krb5kdc[13998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:53.729504 14011 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:04:53.730532 14011 webserver.cc:460] Webserver started at http://127.0.98.62:37339/ using document root <none> and password file <none>
I20260927 15:04:53.731117 14011 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:53.731179 14011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:53.731424 14011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:53.733722 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "bd02894cc69e4ee78cd42c771915f47b"
format_stamp: "Formatted at 2026-09-27 15:04:53 on dist-test-slave-ml7t"
server_key: "72397df91280fa253bbc8904c9fb3896"
server_key_iv: "41281162b9f7c44f12fafd2f75a305aa"
server_key_version: "encryptionkey@0"
I20260927 15:04:53.734560 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd02894cc69e4ee78cd42c771915f47b"
format_stamp: "Formatted at 2026-09-27 15:04:53 on dist-test-slave-ml7t"
server_key: "72397df91280fa253bbc8904c9fb3896"
server_key_iv: "41281162b9f7c44f12fafd2f75a305aa"
server_key_version: "encryptionkey@0"
I20260927 15:04:53.738433 14011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 15:04:53.740866 14026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:53.742264 14011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:04:53.742409 14011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "bd02894cc69e4ee78cd42c771915f47b"
format_stamp: "Formatted at 2026-09-27 15:04:53 on dist-test-slave-ml7t"
server_key: "72397df91280fa253bbc8904c9fb3896"
server_key_iv: "41281162b9f7c44f12fafd2f75a305aa"
server_key_version: "encryptionkey@0"
I20260927 15:04:53.742527 14011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:53.759407 14011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:53.762391 14011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:53.762611 14011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:53.770182 14011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33597
I20260927 15:04:53.770202 14078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33597 every 8 connection(s)
I20260927 15:04:53.771193 14011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:04:53.774008 14079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:53.778790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14011
I20260927 15:04:53.778930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:04:53.779208 392 external_mini_cluster.cc:1428] Setting key 581357d338aad00f1196a32ee3d112bc
I20260927 15:04:53.779743 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b: Bootstrap starting.
I20260927 15:04:53.782327 14079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:53.783070 14079 log.cc:824] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:53.784871 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b: No bootstrap required, opened a new log
Sep 27 15:04:53 dist-test-slave-ml7t krb5kdc[13998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:53.788722 14079 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd02894cc69e4ee78cd42c771915f47b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260927 15:04:53.789011 14079 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:53.789099 14079 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd02894cc69e4ee78cd42c771915f47b, State: Initialized, Role: FOLLOWER
I20260927 15:04:53.789618 14079 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd02894cc69e4ee78cd42c771915f47b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260927 15:04:53.789825 14079 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:04:53.789911 14079 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:04:53.790015 14079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:53.791162 14079 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd02894cc69e4ee78cd42c771915f47b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260927 15:04:53.791524 14079 leader_election.cc:302] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd02894cc69e4ee78cd42c771915f47b; no voters:
I20260927 15:04:53.791872 14079 leader_election.cc:288] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:04:53.791990 14084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:04:53.792225 14084 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [term 1 LEADER]: Becoming Leader. State: Replica: bd02894cc69e4ee78cd42c771915f47b, State: Running, Role: LEADER
I20260927 15:04:53.792596 14084 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd02894cc69e4ee78cd42c771915f47b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260927 15:04:53.793227 14079 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:53.794435 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:53.780573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.0.98.62:33597)
0927 15:04:53.781026 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:53.781041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:53.781076 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:04:53.781721 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:53.781738 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:53.782149 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:53.782500 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:53.783379 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:53.784341 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:53.785259 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:53.785728 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:53.785998 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:53.788524 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:53.788530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:53.788546 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:53.788577 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:53.790745 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:53.791161 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:53.791167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:53.791175 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:53.791315 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:53.791608 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:53.791611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:53.791613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:53.792165 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:53.792939 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:53.793086 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 15:04:53.793378 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260927 15:04:53.794620 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd02894cc69e4ee78cd42c771915f47b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd02894cc69e4ee78cd42c771915f47b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } } }
I20260927 15:04:53.794713 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:53.794421 14086 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd02894cc69e4ee78cd42c771915f47b. Latest consensus state: current_term: 1 leader_uuid: "bd02894cc69e4ee78cd42c771915f47b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd02894cc69e4ee78cd42c771915f47b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } } }
I20260927 15:04:53.794999 14086 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:53.795068 14094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:04:53.797511 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:04:53.798290 14094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:04:53.803736 14094 catalog_manager.cc:1347] Generated new cluster ID: 5cfb29e005a140f288965decdf7ae4d0
I20260927 15:04:53.803815 14094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:04:53.810259 14094 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:04:53.811455 14094 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:04:53.815846 14094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd02894cc69e4ee78cd42c771915f47b: Generated new TSK 0
I20260927 15:04:53.816572 14094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:04:53.853613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14011
2026-09-27T15:04:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1450 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:04:53 dist-test-slave-ml7t krb5kdc[14106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:04:53 dist-test-slave-ml7t krb5kdc[14106](info): set up 2 sockets
Sep 27 15:04:53 dist-test-slave-ml7t krb5kdc[14106](info): commencing operation
krb5kdc: starting...
W20260927 15:04:54.898485 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:04:54 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:04:55.026036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:55.136658 14119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:55.136940 14119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:55.136982 14119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:55.141209 14119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:55.141273 14119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:55.141291 14119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:55.141309 14119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:55.141324 14119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:55.146718 14119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:55.147799 14119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:55.148875 14119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:55.157392 14124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:55.157438 14125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.157429 14119 server_base.cc:1034] running on GCE node
W20260927 15:04:55.157618 14127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.158222 14119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:55.159291 14119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:55.160451 14119 hybrid_clock.cc:648] HybridClock initialized: now 1790521495160435 us; error 29 us; skew 500 ppm
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:55.163395 14119 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:04:55.164357 14119 webserver.cc:460] Webserver started at http://127.0.98.62:33739/ using document root <none> and password file <none>
I20260927 15:04:55.164948 14119 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:55.165060 14119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:55.165316 14119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:55.167506 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "8c5bbfe33a8440e7aec3b580038da264"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "6df9383050e3a970ec60dc6fe157388b"
server_key_iv: "e2671032752fc70f5b9b965990747857"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.168119 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c5bbfe33a8440e7aec3b580038da264"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "6df9383050e3a970ec60dc6fe157388b"
server_key_iv: "e2671032752fc70f5b9b965990747857"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.172207 14119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 15:04:55.174813 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.176045 14119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:04:55.176184 14119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "8c5bbfe33a8440e7aec3b580038da264"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "6df9383050e3a970ec60dc6fe157388b"
server_key_iv: "e2671032752fc70f5b9b965990747857"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.176307 14119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:55.190544 14119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:55.193626 14119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:55.193871 14119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:55.200747 14119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34171
I20260927 15:04:55.201040 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34171 every 8 connection(s)
I20260927 15:04:55.201792 14119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:04:55.202375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14119
I20260927 15:04:55.202517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:04:55.202808 392 external_mini_cluster.cc:1428] Setting key 47d3121a7ac9835ac64af645cb7d12a1
I20260927 15:04:55.204879 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:55.211385 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Bootstrap starting.
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:55.214191 14187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:55.215072 14187 log.cc:824] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:55.217635 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: No bootstrap required, opened a new log
I20260927 15:04:55.218341 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.204275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.0.98.62:34171)
0927 15:04:55.204677 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.204705 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.204742 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:04:55.205433 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.205451 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.205847 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.206133 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:55.207006 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.208000 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.209145 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.209605 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.209737 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.212609 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:55.212616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:55.212636 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:55.212667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:55.214806 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.215321 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.215327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.215333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.215543 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.215987 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.215989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.215990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.216521 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:55.216706 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.216997 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.217306 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":161,"threads_started":1}
I20260927 15:04:55.221827 14187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:55.222070 14187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:55.222107 14187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5bbfe33a8440e7aec3b580038da264, State: Initialized, Role: FOLLOWER
I20260927 15:04:55.222478 14187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:55.222587 14187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:04:55.222626 14187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:04:55.222697 14187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:55.223600 14187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:55.223906 14187 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c5bbfe33a8440e7aec3b580038da264; no voters:
I20260927 15:04:55.224159 14187 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:04:55.224318 14192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:04:55.224576 14192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 LEADER]: Becoming Leader. State: Replica: 8c5bbfe33a8440e7aec3b580038da264, State: Running, Role: LEADER
I20260927 15:04:55.224982 14192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:55.225411 14187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:55.226729 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c5bbfe33a8440e7aec3b580038da264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260927 15:04:55.226852 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:55.227255 14202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:04:55.227236 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5bbfe33a8440e7aec3b580038da264. Latest consensus state: current_term: 1 leader_uuid: "8c5bbfe33a8440e7aec3b580038da264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260927 15:04:55.228281 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:55.230279 14202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:04:55.235816 14202 catalog_manager.cc:1347] Generated new cluster ID: 69a099b022cb4d2cb708026f4fafc19b
I20260927 15:04:55.235893 14202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:04:55.250097 14202 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:04:55.251083 14202 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:04:55.266103 14202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Generated new TSK 0
I20260927 15:04:55.266814 14202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:04:55.302971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:55.412019 14213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:55.412308 14213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:55.412388 14213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:55.416450 14213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:55.416505 14213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:55.416620 14213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:04:55.422382 14213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:55.423451 14213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:55.424466 14213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:55.434129 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:55.434119 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.434235 14213 server_base.cc:1034] running on GCE node
W20260927 15:04:55.434118 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.434782 14213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:55.435458 14213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:55.436673 14213 hybrid_clock.cc:648] HybridClock initialized: now 1790521495436653 us; error 30 us; skew 500 ppm
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:55.439430 14213 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:04:55.440306 14213 webserver.cc:460] Webserver started at http://127.0.98.1:37593/ using document root <none> and password file <none>
I20260927 15:04:55.440851 14213 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:55.440937 14213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:55.441166 14213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:55.443176 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c51e814a0ef466abc5d7cd90d020d98"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "632584e2de9e0539dc3175cd4e3b5de0"
server_key_iv: "ed6f239e4eef174fb9e229245a586fb6"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.443738 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c51e814a0ef466abc5d7cd90d020d98"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "632584e2de9e0539dc3175cd4e3b5de0"
server_key_iv: "ed6f239e4eef174fb9e229245a586fb6"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.447482 14213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:04:55.449976 14228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.451268 14213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:04:55.451401 14213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "8c51e814a0ef466abc5d7cd90d020d98"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "632584e2de9e0539dc3175cd4e3b5de0"
server_key_iv: "ed6f239e4eef174fb9e229245a586fb6"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.451519 14213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:55.471107 14213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:55.474174 14213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:55.474375 14213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:55.474968 14213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:55.475781 14213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:55.475853 14213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.475926 14213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:55.475971 14213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.485071 14213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42599
I20260927 15:04:55.485092 14341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42599 every 8 connection(s)
I20260927 15:04:55.486019 14213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:04:55.489313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14213
I20260927 15:04:55.489452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:04:55.489759 392 external_mini_cluster.cc:1428] Setting key 490faec8f4b42f13f61b5fe7641177ca
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:04:55.500065 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.487888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49037 (local address 127.0.98.62:34171)
0927 15:04:55.488022 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.488026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.489003 (+ 977us) server_negotiation.cc:407] Connection header received
0927 15:04:55.490045 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.490048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.490108 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.490216 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:55.491559 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.492403 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.493581 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.493881 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.493995 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.496466 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:55.496469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:55.496472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:55.496504 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:55.498014 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.498529 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.498537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.498539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.498611 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.498964 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.498969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.498970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.499149 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:55.499262 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.499766 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.499899 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:04:55.501008 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.488211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:49037)
0927 15:04:55.488851 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:55.488892 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:55.489762 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:55.490450 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:55.490457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:55.490809 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:55.491411 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.491426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.492531 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.492534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.493449 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.493458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.494032 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.494036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.494221 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.494848 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:55.494864 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:55.496305 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:55.498127 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:55.498133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:55.498148 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:55.498415 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:55.498709 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:55.498712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:55.498714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:55.498881 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:55.499262 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:55.499270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:55.499527 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 15:04:55.499720 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 15:04:55.500095 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:04:55.502410 14342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260927 15:04:55.502684 14342 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:55.503333 14342 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260927 15:04:55.505087 14151 ts_manager.cc:194] Registered new tserver with Master: 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1:42599)
I20260927 15:04:55.506630 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:04:55.519083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:55.634837 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:55.635118 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:55.635192 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:55.639209 14347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:55.639293 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:55.639382 14347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:04:55.644681 14347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:55.645717 14347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:55.646787 14347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:55.656580 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:55.656594 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.656776 14347 server_base.cc:1034] running on GCE node
W20260927 15:04:55.656661 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.657171 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:55.657862 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:55.659032 14347 hybrid_clock.cc:648] HybridClock initialized: now 1790521495659005 us; error 37 us; skew 500 ppm
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:55.661940 14347 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:04:55.662753 14347 webserver.cc:460] Webserver started at http://127.0.98.2:43153/ using document root <none> and password file <none>
I20260927 15:04:55.663303 14347 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:55.663395 14347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:55.663606 14347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:55.665663 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ca123d85e2145d4b952652d75416338"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "2260fc4b261cf239104bf25f8dbae685"
server_key_iv: "3b44b4effd3bf211d37ea1d50051f589"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.666271 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ca123d85e2145d4b952652d75416338"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "2260fc4b261cf239104bf25f8dbae685"
server_key_iv: "3b44b4effd3bf211d37ea1d50051f589"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.669910 14347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:04:55.672448 14362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.673720 14347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:04:55.673877 14347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "0ca123d85e2145d4b952652d75416338"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "2260fc4b261cf239104bf25f8dbae685"
server_key_iv: "3b44b4effd3bf211d37ea1d50051f589"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.673993 14347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:55.694826 14347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:55.698412 14347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:55.698642 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:55.699244 14347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:55.700109 14347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:55.700182 14347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.700254 14347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:55.700309 14347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.709836 14347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33419
I20260927 15:04:55.709837 14475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33419 every 8 connection(s)
I20260927 15:04:55.710841 14347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:04:55.715668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14347
I20260927 15:04:55.715770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:04:55.716064 392 external_mini_cluster.cc:1428] Setting key 084ad6610c36d8133a61d875a790ccaf
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:55.725176 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.712769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37241 (local address 127.0.98.62:34171)
0927 15:04:55.712920 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.712924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.714020 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 15:04:55.714946 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.714949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.715006 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.715117 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:55.716419 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.717025 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.718774 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.719008 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.719113 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.721679 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:55.721683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:55.721685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:55.721706 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:55.723238 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.723746 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.723749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.723750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.723820 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.724145 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.724150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.724151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.724322 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:55.724432 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.724893 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.725007 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:04:55.726114 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.713177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:37241)
0927 15:04:55.713871 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:55.713914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:55.714732 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:55.715299 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:55.715305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:55.715648 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:55.716262 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.716283 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.717160 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.717163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.718636 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.718650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.719168 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.719173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.719415 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.720078 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:55.720093 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:55.721513 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:55.723353 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:55.723358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:55.723372 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:55.723618 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:55.723912 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:55.723916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:55.723918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:55.724050 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:55.724429 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:55.724435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:55.724667 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 15:04:55.724852 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 15:04:55.725148 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":121,"threads_started":1}
I20260927 15:04:55.727226 14476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260927 15:04:55.727466 14476 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:55.727933 14476 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:04:55.729101 14151 ts_manager.cc:194] Registered new tserver with Master: 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419)
I20260927 15:04:55.729939 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:04:55.745256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:55.860080 14481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:55.860401 14481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:55.860489 14481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:55.864631 14481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:55.864729 14481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:55.864843 14481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:04:55.870293 14481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:55.871376 14481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:55.872582 14481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:55.882740 14487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.882886 14481 server_base.cc:1034] running on GCE node
W20260927 15:04:55.882761 14486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:55.882756 14489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:55.883460 14481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:55.884104 14481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:55.885293 14481 hybrid_clock.cc:648] HybridClock initialized: now 1790521495885278 us; error 27 us; skew 500 ppm
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:55.887977 14481 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:04:55.888780 14481 webserver.cc:460] Webserver started at http://127.0.98.3:34449/ using document root <none> and password file <none>
I20260927 15:04:55.889284 14481 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:55.889341 14481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:55.889499 14481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:04:55.891601 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "36c453d1f2464b2993eb76e556f6178f"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "aefddf8f820caf5f7f123c508c762837"
server_key_iv: "d019e251c2ab06cf1d6ae5178726d5d0"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.892093 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "36c453d1f2464b2993eb76e556f6178f"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "aefddf8f820caf5f7f123c508c762837"
server_key_iv: "d019e251c2ab06cf1d6ae5178726d5d0"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.895627 14481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:04:55.898002 14496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.899245 14481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:04:55.899374 14481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "36c453d1f2464b2993eb76e556f6178f"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "aefddf8f820caf5f7f123c508c762837"
server_key_iv: "d019e251c2ab06cf1d6ae5178726d5d0"
server_key_version: "encryptionkey@0"
I20260927 15:04:55.899488 14481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:55.917490 14481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:55.920617 14481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:55.920823 14481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:55.921461 14481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:55.922307 14481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:55.922353 14481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.922392 14481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:55.922448 14481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:55.931522 14481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39615
I20260927 15:04:55.931550 14609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39615 every 8 connection(s)
I20260927 15:04:55.932513 14481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:04:55.941571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14481
I20260927 15:04:55.941686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
Sep 27 15:04:55 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:55.942024 392 external_mini_cluster.cc:1428] Setting key 84d7f5a5a82685755538167aa65c021d
I20260927 15:04:55.946389 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.934527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40119 (local address 127.0.98.62:34171)
0927 15:04:55.934668 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.934672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.935732 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 15:04:55.936683 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.936687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.936782 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.936879 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:55.938095 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.938623 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.939772 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.939995 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.940107 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.942703 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:55.942707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:55.942711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:55.942736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:55.944386 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.944896 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.944899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.944900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.944981 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.945312 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.945315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.945317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.945516 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:55.945625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.946162 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.946270 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:04:55.947259 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.934882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:40119)
0927 15:04:55.935550 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:55.935593 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:55.936459 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:55.937055 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:55.937061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:55.937427 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:55.937944 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.937958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.938772 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.938779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.939642 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.939651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.940145 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.940149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.940320 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.940894 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:55.940910 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:55.942505 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:55.944507 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:55.944512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:55.944526 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:55.944788 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:55.945066 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:55.945068 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:55.945070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:55.945193 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:55.945624 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:55.945628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:55.945910 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 15:04:55.946112 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 15:04:55.946422 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":130,"threads_started":1}
I20260927 15:04:55.948379 14610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260927 15:04:55.948669 14610 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:55.949182 14610 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260927 15:04:55.950323 14151 ts_manager.cc:194] Registered new tserver with Master: 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
I20260927 15:04:55.950860 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40119
I20260927 15:04:55.956316 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:04:55.966071 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.958393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.98.62:34171)
0927 15:04:55.958513 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.958517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.958751 (+ 234us) server_negotiation.cc:407] Connection header received
0927 15:04:55.958835 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.958837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.958874 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.958942 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:55.959776 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.960280 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.961208 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.961418 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.961500 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.962988 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:55.962991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:55.962993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:55.963014 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:55.964378 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.964768 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.964770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.964771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.964843 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:55.965076 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:55.965078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:55.965079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:55.965295 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:55.965442 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.965735 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.965897 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:04:55.969166 14151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:04:55.972421 14151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:04:55.987418 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.982445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.0.98.3:39615)
0927 15:04:55.982784 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.982795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.982818 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:04:55.982893 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.982898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.983107 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.983350 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:55.983509 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.984390 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.986069 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.986741 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.986872 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.987159 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.987209 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.987297 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":168,"threads_started":1}
I20260927 15:04:55.987406 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.981400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39615 (local address 127.0.0.1:36448)
0927 15:04:55.982269 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:55.982321 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:55.982454 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:55.983260 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:55.983263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:55.983279 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:55.983396 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.983404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.984525 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.984529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.985936 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.985947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.986868 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.986872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.986972 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.987010 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:04:55.987124 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:04:55.987248 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":45,"spinlock_wait_cycles":39296,"thread_start_us":113,"threads_started":1}
I20260927 15:04:55.987855 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.981897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33419 (local address 127.0.0.1:35566)
0927 15:04:55.982269 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:55.982323 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:55.982455 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:55.983616 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:55.983619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:55.983635 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:55.983721 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.983726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.985083 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.985089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.986177 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.986189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.987404 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.987407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.987531 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.987575 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 15:04:55.987649 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:04:55.987723 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":40,"thread_start_us":101,"threads_started":1}
I20260927 15:04:55.987977 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.981470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.0.98.1:42599)
0927 15:04:55.981761 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.981797 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.982409 (+ 612us) server_negotiation.cc:407] Connection header received
0927 15:04:55.982630 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.982635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.982799 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.982927 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:55.983617 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.984803 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.986371 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.987107 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.987208 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.987339 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.987726 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.987822 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260927 15:04:55.988348 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.982053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.0.98.2:33419)
0927 15:04:55.983040 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:55.983050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:55.983071 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:04:55.983138 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:55.983142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:55.983315 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:55.983445 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:55.983850 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.984952 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.986280 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.987088 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.987217 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.987339 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:55.988094 (+ 755us) server_negotiation.cc:299] Negotiation successful
0927 15:04:55.988196 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":146,"threads_started":1}
I20260927 15:04:55.988454 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:55.981330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42599 (local address 127.0.0.1:43574)
0927 15:04:55.982269 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:55.982323 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:55.982484 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:55.983147 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:55.983152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:55.983174 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:55.983514 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.983518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.984940 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.984943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.986211 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:55.986219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:55.987404 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:55.987407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:55.987554 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:55.987591 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:04:55.988251 (+ 660us) client_negotiation.cc:240] Negotiation successful
0927 15:04:55.988320 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":198,"thread_start_us":86,"threads_started":1}
I20260927 15:04:55.990918 14276 tablet_service.cc:1467] Processing CreateTablet for tablet d01cd0f96d944aefbb06f84d95d29fc5 (DEFAULT_TABLE table=test-table [id=84d99f5c84f74fd0b1149121d27cb8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:55.990918 14544 tablet_service.cc:1467] Processing CreateTablet for tablet d01cd0f96d944aefbb06f84d95d29fc5 (DEFAULT_TABLE table=test-table [id=84d99f5c84f74fd0b1149121d27cb8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:55.990918 14410 tablet_service.cc:1467] Processing CreateTablet for tablet d01cd0f96d944aefbb06f84d95d29fc5 (DEFAULT_TABLE table=test-table [id=84d99f5c84f74fd0b1149121d27cb8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:55.991935 14276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01cd0f96d944aefbb06f84d95d29fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:55.991931 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01cd0f96d944aefbb06f84d95d29fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:55.991940 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01cd0f96d944aefbb06f84d95d29fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:55.999187 14627 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap starting.
I20260927 15:04:55.999521 14629 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Bootstrap starting.
I20260927 15:04:55.999791 14628 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Bootstrap starting.
I20260927 15:04:56.001291 14627 tablet_bootstrap.cc:654] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:56.001685 14628 tablet_bootstrap.cc:654] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:56.002207 14629 tablet_bootstrap.cc:654] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:56.002539 14627 log.cc:824] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:56.003021 14629 log.cc:824] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:56.003021 14628 log.cc:824] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:56.004917 14627 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: No bootstrap required, opened a new log
I20260927 15:04:56.004936 14628 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: No bootstrap required, opened a new log
I20260927 15:04:56.004937 14629 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: No bootstrap required, opened a new log
I20260927 15:04:56.005137 14628 ts_tablet_manager.cc:1397] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:04:56.005137 14629 ts_tablet_manager.cc:1397] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:04:56.005144 14627 ts_tablet_manager.cc:1397] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:04:56.008147 14627 raft_consensus.cc:348] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.008457 14627 raft_consensus.cc:374] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:56.008530 14627 raft_consensus.cc:729] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51e814a0ef466abc5d7cd90d020d98, State: Initialized, Role: FOLLOWER
I20260927 15:04:56.009063 14627 consensus_queue.cc:260] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.009087 14629 raft_consensus.cc:348] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.009092 14628 raft_consensus.cc:348] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.009361 14628 raft_consensus.cc:374] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:56.009361 14629 raft_consensus.cc:374] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:56.009420 14629 raft_consensus.cc:729] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca123d85e2145d4b952652d75416338, State: Initialized, Role: FOLLOWER
I20260927 15:04:56.009428 14628 raft_consensus.cc:729] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c453d1f2464b2993eb76e556f6178f, State: Initialized, Role: FOLLOWER
I20260927 15:04:56.009858 14628 consensus_queue.cc:260] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.009891 14629 consensus_queue.cc:260] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.010183 14627 ts_tablet_manager.cc:1428] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 15:04:56.010319 14342 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260927 15:04:56.010574 14476 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260927 15:04:56.010840 14629 ts_tablet_manager.cc:1428] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:04:56.012104 14628 ts_tablet_manager.cc:1428] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 15:04:56.012435 14610 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260927 15:04:56.087222 14635 raft_consensus.cc:482] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:04:56.087481 14635 raft_consensus.cc:504] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.088744 14635 leader_election.cc:288] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1:42599), 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419)
I20260927 15:04:56.092459 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.089257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36267 (local address 127.0.98.2:33419)
0927 15:04:56.089393 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.089396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.089407 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:04:56.089454 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.089456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.089495 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.089606 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:56.089944 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.090459 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.091482 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.091895 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.092015 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.092080 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.092263 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.092344 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:04:56.092459 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.089017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33419 (local address 127.0.98.3:36267)
0927 15:04:56.089143 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:56.089160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:56.089255 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:56.089619 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:56.089622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:56.089639 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:56.089817 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.089827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.090572 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.090578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.091353 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.091361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.092017 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.092021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.092140 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.092158 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:04:56.092246 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:04:56.092325 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 15:04:56.092665 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.089171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59957 (local address 127.0.98.1:42599)
0927 15:04:56.089299 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.089305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.089558 (+ 253us) server_negotiation.cc:407] Connection header received
0927 15:04:56.089710 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.089714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.089807 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.089900 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:56.090172 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.090682 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.091710 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.092102 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.092219 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.092280 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.092483 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.092550 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:04:56.092666 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.089017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42599 (local address 127.0.98.3:59957)
0927 15:04:56.089460 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:56.089471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:56.089595 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:56.089919 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:56.089922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:56.089938 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:56.090055 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.090061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.090795 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.090799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.091567 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.091575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.092221 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.092225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.092342 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.092369 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:04:56.092455 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:04:56.092521 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":30,"thread_start_us":128,"threads_started":1}
I20260927 15:04:56.093257 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01cd0f96d944aefbb06f84d95d29fc5" candidate_uuid: "36c453d1f2464b2993eb76e556f6178f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c51e814a0ef466abc5d7cd90d020d98" is_pre_election: true
I20260927 15:04:56.093251 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01cd0f96d944aefbb06f84d95d29fc5" candidate_uuid: "36c453d1f2464b2993eb76e556f6178f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca123d85e2145d4b952652d75416338" is_pre_election: true
I20260927 15:04:56.093596 14296 raft_consensus.cc:2454] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c453d1f2464b2993eb76e556f6178f in term 0.
I20260927 15:04:56.093611 14430 raft_consensus.cc:2454] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c453d1f2464b2993eb76e556f6178f in term 0.
I20260927 15:04:56.094164 14497 leader_election.cc:302] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36c453d1f2464b2993eb76e556f6178f, 8c51e814a0ef466abc5d7cd90d020d98; no voters:
I20260927 15:04:56.094437 14635 raft_consensus.cc:2781] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:04:56.094551 14635 raft_consensus.cc:482] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:04:56.094611 14635 raft_consensus.cc:3037] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:56.095626 14635 raft_consensus.cc:504] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.095993 14635 leader_election.cc:288] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [CANDIDATE]: Term 1 election: Requested vote from peers 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1:42599), 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419)
I20260927 15:04:56.096382 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01cd0f96d944aefbb06f84d95d29fc5" candidate_uuid: "36c453d1f2464b2993eb76e556f6178f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c51e814a0ef466abc5d7cd90d020d98"
I20260927 15:04:56.096382 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01cd0f96d944aefbb06f84d95d29fc5" candidate_uuid: "36c453d1f2464b2993eb76e556f6178f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca123d85e2145d4b952652d75416338"
I20260927 15:04:56.096511 14430 raft_consensus.cc:3037] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:56.096511 14296 raft_consensus.cc:3037] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:56.097839 14430 raft_consensus.cc:2454] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c453d1f2464b2993eb76e556f6178f in term 1.
I20260927 15:04:56.097839 14296 raft_consensus.cc:2454] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c453d1f2464b2993eb76e556f6178f in term 1.
I20260927 15:04:56.098143 14497 leader_election.cc:302] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36c453d1f2464b2993eb76e556f6178f, 8c51e814a0ef466abc5d7cd90d020d98; no voters:
I20260927 15:04:56.098368 14635 raft_consensus.cc:2781] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:04:56.098606 14635 raft_consensus.cc:686] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 1 LEADER]: Becoming Leader. State: Replica: 36c453d1f2464b2993eb76e556f6178f, State: Running, Role: LEADER
I20260927 15:04:56.098954 14635 consensus_queue.cc:237] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.102035 14151 catalog_manager.cc:5495] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36c453d1f2464b2993eb76e556f6178f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "36c453d1f2464b2993eb76e556f6178f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } health_report { overall_health: UNKNOWN } } }
I20260927 15:04:56.140939 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.136990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.0.98.3:39615)
0927 15:04:56.137156 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.137161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.137178 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:04:56.137374 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.137376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.137448 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.137578 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:04:56.138030 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.138539 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.139439 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.139661 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.139777 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.140097 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:04:56.140525 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:04:56.140665 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.140708 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.140767 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:04:56.146718 14296 raft_consensus.cc:1261] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Refusing update from remote peer 36c453d1f2464b2993eb76e556f6178f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:04:56.146708 14430 raft_consensus.cc:1261] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Refusing update from remote peer 36c453d1f2464b2993eb76e556f6178f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:04:56.147469 14637 consensus_queue.cc:1035] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:56.147677 14635 consensus_queue.cc:1035] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:56.153370 14640 mvcc.cc:205] Tried to move back new op lower bound from 7333976048209981440 to 7333976048023871488. Current Snapshot: MvccSnapshot[applied={T|T < 7333976048209981440}]
I20260927 15:04:56.155431 14642 mvcc.cc:205] Tried to move back new op lower bound from 7333976048209981440 to 7333976048023871488. Current Snapshot: MvccSnapshot[applied={T|T < 7333976048209981440}]
I20260927 15:04:56.155869 14641 mvcc.cc:205] Tried to move back new op lower bound from 7333976048209981440 to 7333976048023871488. Current Snapshot: MvccSnapshot[applied={T|T < 7333976048209981440}]
I20260927 15:04:56.162662 14150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:04:56.162801 14150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:04:56.165267 14150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Sending DeleteTablet for 3 replicas of tablet d01cd0f96d944aefbb06f84d95d29fc5
I20260927 15:04:56.166079 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet d01cd0f96d944aefbb06f84d95d29fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36448
I20260927 15:04:56.166061 14276 tablet_service.cc:1514] Processing DeleteTablet for tablet d01cd0f96d944aefbb06f84d95d29fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43574
I20260927 15:04:56.166237 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet d01cd0f96d944aefbb06f84d95d29fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35566
I20260927 15:04:56.166628 14649 tablet_replica.cc:330] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: stopping tablet replica
I20260927 15:04:56.166836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14213
I20260927 15:04:56.166860 14650 tablet_replica.cc:330] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: stopping tablet replica
I20260927 15:04:56.166951 14649 raft_consensus.cc:2229] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:04:56.167146 14650 raft_consensus.cc:2229] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:04:56.167346 14649 raft_consensus.cc:2258] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:04:56.167506 14651 tablet_replica.cc:330] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: stopping tablet replica
I20260927 15:04:56.167786 14651 raft_consensus.cc:2229] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:04:56.168046 14651 raft_consensus.cc:2258] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:04:56.168895 14649 ts_tablet_manager.cc:1905] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:56.169494 14651 ts_tablet_manager.cc:1905] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:56.171657 14649 ts_tablet_manager.cc:1918] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:56.171838 14649 log.cc:1196] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/d01cd0f96d944aefbb06f84d95d29fc5
I20260927 15:04:56.172376 14649 ts_tablet_manager.cc:1939] T d01cd0f96d944aefbb06f84d95d29fc5 P 36c453d1f2464b2993eb76e556f6178f: Deleting consensus metadata
I20260927 15:04:56.173205 14651 ts_tablet_manager.cc:1918] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:56.173367 14651 log.cc:1196] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/d01cd0f96d944aefbb06f84d95d29fc5
I20260927 15:04:56.173684 14136 catalog_manager.cc:4841] TS 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615): tablet d01cd0f96d944aefbb06f84d95d29fc5 (table test-table [id=84d99f5c84f74fd0b1149121d27cb8eb]) successfully deleted
I20260927 15:04:56.173729 14651 ts_tablet_manager.cc:1939] T d01cd0f96d944aefbb06f84d95d29fc5 P 0ca123d85e2145d4b952652d75416338: Deleting consensus metadata
I20260927 15:04:56.174934 14138 catalog_manager.cc:4841] TS 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419): tablet d01cd0f96d944aefbb06f84d95d29fc5 (table test-table [id=84d99f5c84f74fd0b1149121d27cb8eb]) successfully deleted
W20260927 15:04:56.175998 14135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42599 (error 108)
I20260927 15:04:56.176785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14347
I20260927 15:04:56.177039 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.176741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42599 (local address 127.0.0.1:43586)
0927 15:04:56.176872 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:56.176943 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:04:56.178478 14135 catalog_manager.cc:4584] TS 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1:42599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d01cd0f96d944aefbb06f84d95d29fc5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42599: connect: Connection refused (error 111)
I20260927 15:04:56.183162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14481
I20260927 15:04:56.188966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14119
I20260927 15:04:56.195438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--webserver_interface=127.0.98.62
--webserver_port=33739
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:56.305663 14653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:56.305938 14653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:56.305976 14653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:56.309898 14653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:56.309957 14653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:56.309978 14653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:56.309993 14653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:56.310009 14653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:56.315238 14653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:56.316291 14653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:56.317349 14653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:56.330448 14659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:56.330466 14662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:56.330588 14660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:56.330925 14653 server_base.cc:1034] running on GCE node
I20260927 15:04:56.331530 14653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:56.332690 14653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:56.333871 14653 hybrid_clock.cc:648] HybridClock initialized: now 1790521496333855 us; error 26 us; skew 500 ppm
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:56.336746 14653 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:04:56.337687 14653 webserver.cc:460] Webserver started at http://127.0.98.62:33739/ using document root <none> and password file <none>
I20260927 15:04:56.338322 14653 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:56.338438 14653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:56.342445 14653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:04:56.344686 14669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.346374 14653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:04:56.346506 14653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "8c5bbfe33a8440e7aec3b580038da264"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "6df9383050e3a970ec60dc6fe157388b"
server_key_iv: "e2671032752fc70f5b9b965990747857"
server_key_version: "encryptionkey@0"
I20260927 15:04:56.347114 14653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:56.365407 14653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:56.368270 14653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:56.368479 14653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:56.376187 14653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34171
I20260927 15:04:56.376475 14721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34171 every 8 connection(s)
I20260927 15:04:56.377331 14653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:04:56.381649 14722 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:04:56.382042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14653
I20260927 15:04:56.382453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42599
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=37593
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:56.383473 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Bootstrap starting.
I20260927 15:04:56.400578 14722 log.cc:824] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:56.409341 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:04:56.409807 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Bootstrap complete.
I20260927 15:04:56.413867 14722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:56.414281 14722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5bbfe33a8440e7aec3b580038da264, State: Initialized, Role: FOLLOWER
I20260927 15:04:56.414739 14722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:56.414875 14722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:04:56.414963 14722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:04:56.415084 14722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:04:56.416870 14722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:56.417194 14722 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c5bbfe33a8440e7aec3b580038da264; no voters:
I20260927 15:04:56.417512 14722 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:04:56.417625 14726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:04:56.417858 14726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 LEADER]: Becoming Leader. State: Replica: 8c5bbfe33a8440e7aec3b580038da264, State: Running, Role: LEADER
I20260927 15:04:56.418203 14726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:56.418725 14722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:56.419091 14726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5bbfe33a8440e7aec3b580038da264. Latest consensus state: current_term: 2 leader_uuid: "8c5bbfe33a8440e7aec3b580038da264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260927 15:04:56.419286 14726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:56.419454 14727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c5bbfe33a8440e7aec3b580038da264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260927 15:04:56.419632 14727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:56.419723 14730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:04:56.424688 14730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:04:56.425556 14730 catalog_manager.cc:1259] Loaded cluster ID: 69a099b022cb4d2cb708026f4fafc19b
I20260927 15:04:56.425637 14730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:04:56.427145 14730 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 15:04:56.432076 14730 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 15:04:56.437821 14730 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:04:56.438642 14730 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Loaded TSK: 0
I20260927 15:04:56.439968 14730 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:04:56.501916 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:56.502192 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:56.502264 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:56.506256 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:56.506331 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:56.506415 14724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:04:56.511754 14724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37593
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:56.512836 14724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:56.513901 14724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:56.524843 14749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:56.524845 14750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:56.525260 14724 server_base.cc:1034] running on GCE node
W20260927 15:04:56.525424 14752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:56.525920 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:56.526636 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:56.527834 14724 hybrid_clock.cc:648] HybridClock initialized: now 1790521496527814 us; error 38 us; skew 500 ppm
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:56.530723 14724 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:04:56.531715 14724 webserver.cc:460] Webserver started at http://127.0.98.1:37593/ using document root <none> and password file <none>
I20260927 15:04:56.532276 14724 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:56.532364 14724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:56.536432 14724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:04:56.538657 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.539983 14724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:04:56.540076 14724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "8c51e814a0ef466abc5d7cd90d020d98"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "632584e2de9e0539dc3175cd4e3b5de0"
server_key_iv: "ed6f239e4eef174fb9e229245a586fb6"
server_key_version: "encryptionkey@0"
I20260927 15:04:56.540632 14724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:56.575711 14724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:56.579025 14724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:56.579241 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:56.579862 14724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:56.581007 14766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:04:56.585083 14724 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:04:56.585168 14724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 15:04:56.585251 14724 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:04:56.586953 14724 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:04:56.587023 14724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 15:04:56.587263 14766 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap starting.
I20260927 15:04:56.597311 14724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42599
I20260927 15:04:56.597357 14873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42599 every 8 connection(s)
I20260927 15:04:56.598295 14724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:04:56.599380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14724
I20260927 15:04:56.599797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33419
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=43153
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:56.607076 14766 log.cc:824] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:56.616477 14766 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:04:56.616549 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.600627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34589 (local address 127.0.98.62:34171)
0927 15:04:56.601127 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.601147 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.602003 (+ 856us) server_negotiation.cc:407] Connection header received
0927 15:04:56.603387 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.603403 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.603798 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.604207 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:56.605545 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.606734 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.608197 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.608654 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.608835 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.611319 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:56.611325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:56.611336 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:56.611362 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:56.613215 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.613756 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.613762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.613797 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.613926 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.614285 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.614290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.614292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.614649 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:56.614801 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.615342 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.615599 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":117,"threads_started":1}
I20260927 15:04:56.616532 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.600559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:34589)
0927 15:04:56.601792 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:56.601855 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:56.602911 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:56.604230 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:56.604237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:56.604576 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:56.605303 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.605331 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.606906 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.606909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.608072 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.608081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.608800 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.608804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.608985 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.609602 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:56.609618 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:56.611116 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:56.613358 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.613363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.613376 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.613636 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.614035 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.614038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.614040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.614187 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.614784 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:56.614790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:56.615038 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 15:04:56.615227 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 15:04:56.615563 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":144,"threads_started":1}
I20260927 15:04:56.617234 14766 tablet_bootstrap.cc:492] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap complete.
I20260927 15:04:56.618691 14766 ts_tablet_manager.cc:1397] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.015s
I20260927 15:04:56.619352 14874 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260927 15:04:56.619575 14874 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:56.620165 14874 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260927 15:04:56.621580 14766 raft_consensus.cc:348] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.622144 14766 raft_consensus.cc:729] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51e814a0ef466abc5d7cd90d020d98, State: Initialized, Role: FOLLOWER
I20260927 15:04:56.622684 14686 ts_manager.cc:194] Registered new tserver with Master: 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1:42599)
I20260927 15:04:56.622673 14766 consensus_queue.cc:260] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } }
I20260927 15:04:56.623643 14766 ts_tablet_manager.cc:1428] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 15:04:56.626045 14686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34589
I20260927 15:04:56.628510 14874 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:04:56.637221 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.625540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42599 (local address 127.0.0.1:43600)
0927 15:04:56.626354 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:56.626399 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:56.626516 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:56.627356 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:56.627361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:56.627807 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:56.627934 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.627943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.629858 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.629861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.630771 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.630781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.631321 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.631337 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.631440 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.632003 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:56.632018 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:56.633482 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:56.635512 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.635522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.635528 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.635770 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.636073 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.636077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.636079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.636167 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.636615 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:56.636620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:56.636777 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 15:04:56.636931 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 15:04:56.637092 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":116,"threads_started":1}
I20260927 15:04:56.637355 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.625622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.0.98.1:42599)
0927 15:04:56.626902 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.626914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.626932 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:04:56.626982 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.626986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.627196 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.627331 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:56.628070 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.629037 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.630882 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.631223 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.631308 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.633630 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:56.633639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:56.633648 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:56.633671 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:56.635359 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.635886 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.635892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.635894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.635978 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.636265 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.636268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.636270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.636499 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:56.636639 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.637033 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.637197 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1187,"thread_start_us":115,"threads_started":1}
I20260927 15:04:56.638331 14808 tablet_service.cc:1514] Processing DeleteTablet for tablet d01cd0f96d944aefbb06f84d95d29fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43600
I20260927 15:04:56.639169 14883 tablet_replica.cc:330] stopping tablet replica
I20260927 15:04:56.639457 14883 raft_consensus.cc:2229] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:04:56.639704 14883 raft_consensus.cc:2258] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:04:56.640607 14883 ts_tablet_manager.cc:1905] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:56.644014 14883 ts_tablet_manager.cc:1918] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:56.644145 14883 log.cc:1196] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/d01cd0f96d944aefbb06f84d95d29fc5
I20260927 15:04:56.644398 14883 ts_tablet_manager.cc:1939] T d01cd0f96d944aefbb06f84d95d29fc5 P 8c51e814a0ef466abc5d7cd90d020d98: Deleting consensus metadata
I20260927 15:04:56.646008 14670 catalog_manager.cc:4841] TS 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1:42599): tablet d01cd0f96d944aefbb06f84d95d29fc5 (table test-table [id=84d99f5c84f74fd0b1149121d27cb8eb]) successfully deleted
W20260927 15:04:56.721174 14878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:56.721414 14878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:56.721496 14878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:56.725564 14878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:56.725636 14878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:56.725695 14878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:04:56.731096 14878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43153
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:56.732172 14878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:56.733174 14878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:04:56.742879 14878 server_base.cc:1034] running on GCE node
W20260927 15:04:56.742813 14889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:56.742847 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:56.742833 14888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:56.743489 14878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:56.744184 14878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:56.745357 14878 hybrid_clock.cc:648] HybridClock initialized: now 1790521496745327 us; error 51 us; skew 500 ppm
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:56.748525 14878 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:04:56.749532 14878 webserver.cc:460] Webserver started at http://127.0.98.2:43153/ using document root <none> and password file <none>
I20260927 15:04:56.750208 14878 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:56.750352 14878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:56.754580 14878 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:04:56.756798 14898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.758211 14878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:04:56.758337 14878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "0ca123d85e2145d4b952652d75416338"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "2260fc4b261cf239104bf25f8dbae685"
server_key_iv: "3b44b4effd3bf211d37ea1d50051f589"
server_key_version: "encryptionkey@0"
I20260927 15:04:56.758872 14878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:56.780413 14878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:56.783986 14878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:56.784210 14878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:56.784840 14878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:56.785696 14878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:56.785745 14878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.785864 14878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:56.785905 14878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.795634 14878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33419
I20260927 15:04:56.795651 15011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33419 every 8 connection(s)
I20260927 15:04:56.796694 14878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:04:56.799424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 14878
I20260927 15:04:56.799881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39615
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=34449
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:56.813330 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.798602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53897 (local address 127.0.98.62:34171)
0927 15:04:56.798755 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.798759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.799974 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 15:04:56.800949 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.800956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.801032 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.801181 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:56.802952 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.804016 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.805295 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.805831 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.805982 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.808915 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:56.808923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:56.808930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:56.808975 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:56.810940 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.811480 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.811487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.811493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.811600 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.811972 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.811978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.811982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.812273 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:56.812420 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.813000 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.813144 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:04:56.814266 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.799007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:53897)
0927 15:04:56.799785 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:56.799842 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:56.800720 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:56.801432 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:56.801446 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:56.802028 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:56.802748 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.802769 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.804207 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.804210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.805152 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.805162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.806002 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.806005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.806185 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.806894 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:56.806912 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:56.808724 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:56.811098 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.811103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.811115 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.811367 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.811726 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.811728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.811729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.811866 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.812439 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:56.812444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:56.812720 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 15:04:56.812922 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 15:04:56.813206 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":150,"threads_started":1}
I20260927 15:04:56.815544 15012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260927 15:04:56.815822 15012 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:56.816401 15012 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260927 15:04:56.817512 14686 ts_manager.cc:194] Registered new tserver with Master: 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419)
I20260927 15:04:56.818138 14686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53897
W20260927 15:04:56.917299 15015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:56.917569 15015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:56.917614 15015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:56.922024 15015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:56.922081 15015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:56.922187 15015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:04:56.927649 15015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34449
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:56.928741 15015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:56.929759 15015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:04:56.939898 15015 server_base.cc:1034] running on GCE node
W20260927 15:04:56.939772 15023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:56.939800 15020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:56.939797 15021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:56.940506 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:56.941121 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:56.942307 15015 hybrid_clock.cc:648] HybridClock initialized: now 1790521496942273 us; error 47 us; skew 500 ppm
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:56.945171 15015 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:04:56.946137 15015 webserver.cc:460] Webserver started at http://127.0.98.3:34449/ using document root <none> and password file <none>
I20260927 15:04:56.946656 15015 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:56.946727 15015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:56.950529 15015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:04:56.952771 15030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.954069 15015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:04:56.954188 15015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "36c453d1f2464b2993eb76e556f6178f"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "aefddf8f820caf5f7f123c508c762837"
server_key_iv: "d019e251c2ab06cf1d6ae5178726d5d0"
server_key_version: "encryptionkey@0"
I20260927 15:04:56.954705 15015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:56.969754 15015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:56.972498 15015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:56.972644 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:56.973179 15015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:56.974007 15015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:04:56.974088 15015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.974161 15015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:56.974193 15015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:56.983177 15015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39615
I20260927 15:04:56.983191 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39615 every 8 connection(s)
I20260927 15:04:56.984207 15015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:04:56.987370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15015
Sep 27 15:04:56 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:04:56.996234 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.988026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.0.98.62:34171)
0927 15:04:56.988349 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.988353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.988367 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:04:56.988502 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.988505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.988557 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.988655 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:56.989379 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.989999 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.990846 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.991084 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.991193 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.992896 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:56.992900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:56.992902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:56.992922 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:56.994453 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.994917 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.994920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.994921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.995000 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.995311 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.995314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.995315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.995485 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:56.995626 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.995873 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.996005 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260927 15:04:56.998116 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.986063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55239 (local address 127.0.98.62:34171)
0927 15:04:56.986224 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:56.986227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:56.987280 (+ 1053us) server_negotiation.cc:407] Connection header received
0927 15:04:56.988323 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:56.988327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:56.988380 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:56.988489 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:56.989652 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.990211 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.991336 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.991622 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.991729 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.994361 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:56.994365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:56.994366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:56.994388 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:56.995897 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.996471 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.996475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.996476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.996561 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:56.997004 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:56.997006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:56.997008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:56.997231 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:56.997318 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:56.997856 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 15:04:56.997965 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":768}
I20260927 15:04:56.999032 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:56.986474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:55239)
0927 15:04:56.987127 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:56.987168 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:56.988082 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:56.988668 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:56.988676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:56.989044 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:56.989510 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.989522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.990335 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.990339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.991200 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:56.991209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:56.991776 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:56.991794 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:56.991998 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:56.992616 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:56.992637 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:56.994185 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:56.996007 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.996013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.996027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.996374 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.996678 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:56.996681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:56.996683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:56.996875 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:56.997346 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:56.997352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:56.997598 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 15:04:56.997761 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:04:56.998213 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":111,"threads_started":1}
I20260927 15:04:57.000123 15144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260927 15:04:57.000412 15144 heartbeater.cc:464] Registering TS with master...
I20260927 15:04:57.000950 15144 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260927 15:04:57.002058 14686 ts_manager.cc:194] Registered new tserver with Master: 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
I20260927 15:04:57.002574 14686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55239
I20260927 15:04:57.010529 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:04:57.015333 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.011273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.0.98.62:34171)
0927 15:04:57.011461 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:57.011465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:57.011563 (+ 98us) server_negotiation.cc:407] Connection header received
0927 15:04:57.011695 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:57.011697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:57.011804 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:57.011918 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:04:57.012224 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.012824 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.013733 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.014030 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.014120 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.014449 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:04:57.014900 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:04:57.014995 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:57.015104 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 15:04:57.015156 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:04:57.016080 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:04:57.019531 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:04:57.027446 14808 tablet_service.cc:1467] Processing CreateTablet for tablet 81633a49552143fa83d864c0b69a7f78 (DEFAULT_TABLE table=test-table [id=9b89c32ffef34237a8eb1f7e21933a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:57.028120 14808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81633a49552143fa83d864c0b69a7f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:57.030802 14766 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap starting.
I20260927 15:04:57.031713 14766 tablet_bootstrap.cc:654] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:57.032677 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.027090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39615 (local address 127.0.0.1:36472)
0927 15:04:57.027365 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.027377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:57.027485 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:57.028436 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:57.028440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:57.028458 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:57.028541 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.028547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.030057 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.030060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.031138 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.031148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.032296 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.032299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.032364 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.032398 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:04:57.032449 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 15:04:57.032502 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":126,"threads_started":1}
I20260927 15:04:57.032809 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.027236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.0.98.3:39615)
0927 15:04:57.027587 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:57.027598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:57.027621 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:04:57.027701 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:57.027706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:57.027924 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:57.028088 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:57.028788 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.029922 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.031264 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.032156 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.032279 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.032456 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:57.032510 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 15:04:57.032631 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":132,"threads_started":1}
I20260927 15:04:57.033311 14766 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: No bootstrap required, opened a new log
I20260927 15:04:57.033402 14766 ts_tablet_manager.cc:1397] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 15:04:57.033907 14766 raft_consensus.cc:348] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.034030 14766 raft_consensus.cc:374] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:57.034070 14766 raft_consensus.cc:729] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51e814a0ef466abc5d7cd90d020d98, State: Initialized, Role: FOLLOWER
I20260927 15:04:57.034371 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.026305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33419 (local address 127.0.0.1:35582)
0927 15:04:57.026458 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.026472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:57.026550 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:57.027923 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:57.027926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:57.027944 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:57.028068 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.028093 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.029572 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.029575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.030590 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.030599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.033925 (+ 3326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.033930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.034027 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.034054 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:04:57.034169 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:04:57.034260 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 15:04:57.034332 14766 consensus_queue.cc:260] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.034964 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.026519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.0.98.2:33419)
0927 15:04:57.026967 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:57.026980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:57.027004 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:04:57.027084 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:57.027089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:57.027755 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:57.027933 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:57.028233 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.029410 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.032873 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.033779 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.033907 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.034031 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:57.034704 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 15:04:57.034829 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":126,"threads_started":1}
I20260927 15:04:57.035982 14766 ts_tablet_manager.cc:1428] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:04:57.036371 15078 tablet_service.cc:1467] Processing CreateTablet for tablet 81633a49552143fa83d864c0b69a7f78 (DEFAULT_TABLE table=test-table [id=9b89c32ffef34237a8eb1f7e21933a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:57.037467 15078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81633a49552143fa83d864c0b69a7f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:57.037711 14946 tablet_service.cc:1467] Processing CreateTablet for tablet 81633a49552143fa83d864c0b69a7f78 (DEFAULT_TABLE table=test-table [id=9b89c32ffef34237a8eb1f7e21933a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:04:57.038736 14946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81633a49552143fa83d864c0b69a7f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:04:57.045630 15157 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Bootstrap starting.
I20260927 15:04:57.046167 15158 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Bootstrap starting.
I20260927 15:04:57.048444 15157 tablet_bootstrap.cc:654] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:57.048532 15158 tablet_bootstrap.cc:654] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Neither blocks nor log segments found. Creating new log.
I20260927 15:04:57.049338 15158 log.cc:824] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:57.049337 15157 log.cc:824] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:57.051445 15158 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: No bootstrap required, opened a new log
I20260927 15:04:57.051445 15157 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: No bootstrap required, opened a new log
I20260927 15:04:57.051647 15158 ts_tablet_manager.cc:1397] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:04:57.051652 15157 ts_tablet_manager.cc:1397] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:04:57.055483 15158 raft_consensus.cc:348] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.055483 15157 raft_consensus.cc:348] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.055763 15157 raft_consensus.cc:374] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:57.055763 15158 raft_consensus.cc:374] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:04:57.055816 15158 raft_consensus.cc:729] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca123d85e2145d4b952652d75416338, State: Initialized, Role: FOLLOWER
I20260927 15:04:57.055816 15157 raft_consensus.cc:729] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c453d1f2464b2993eb76e556f6178f, State: Initialized, Role: FOLLOWER
I20260927 15:04:57.056262 15157 consensus_queue.cc:260] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.056262 15158 consensus_queue.cc:260] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.056895 15012 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260927 15:04:57.057228 15157 ts_tablet_manager.cc:1428] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:04:57.057232 15144 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260927 15:04:57.057494 15158 ts_tablet_manager.cc:1428] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:04:57.076769 14880 raft_consensus.cc:482] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:04:57.077015 14880 raft_consensus.cc:504] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.078235 14880 leader_election.cc:288] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419), 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
I20260927 15:04:57.082074 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.078543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35459 (local address 127.0.98.2:33419)
0927 15:04:57.078690 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:57.078695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:57.078711 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:04:57.078865 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:57.078871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:57.078933 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:57.079014 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:57.079349 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.079933 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.081039 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.081462 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.081544 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.081649 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:57.081868 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 15:04:57.081941 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:04:57.082094 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.078630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39963 (local address 127.0.98.3:39615)
0927 15:04:57.078775 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:57.078780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:57.078949 (+ 169us) server_negotiation.cc:407] Connection header received
0927 15:04:57.079071 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:57.079074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:57.079122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:57.079203 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:04:57.079496 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.079965 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.081094 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.081485 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.081611 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.081672 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:57.081922 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 15:04:57.081989 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:04:57.082089 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.078434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33419 (local address 127.0.98.1:35459)
0927 15:04:57.078588 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.078605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:57.078714 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:57.079058 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:57.079061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:57.079080 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:57.079231 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.079239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.080048 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.080054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.080910 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.080919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.081592 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.081595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.081701 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.081758 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 15:04:57.081841 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:04:57.081942 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":3}
I20260927 15:04:57.082114 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.078462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39615 (local address 127.0.98.1:39963)
0927 15:04:57.078847 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.078860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:57.078965 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:57.079224 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:57.079226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:57.079240 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:04:57.079385 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.079391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.080103 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.080105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.080945 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:57.080954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.081649 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.081654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:57.081744 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.081761 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:04:57.081887 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:04:57.081952 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":39,"thread_start_us":82,"threads_started":1}
I20260927 15:04:57.082834 14966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81633a49552143fa83d864c0b69a7f78" candidate_uuid: "8c51e814a0ef466abc5d7cd90d020d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca123d85e2145d4b952652d75416338" is_pre_election: true
I20260927 15:04:57.082942 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81633a49552143fa83d864c0b69a7f78" candidate_uuid: "8c51e814a0ef466abc5d7cd90d020d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c453d1f2464b2993eb76e556f6178f" is_pre_election: true
I20260927 15:04:57.083174 14966 raft_consensus.cc:2454] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c51e814a0ef466abc5d7cd90d020d98 in term 0.
I20260927 15:04:57.083173 15098 raft_consensus.cc:2454] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c51e814a0ef466abc5d7cd90d020d98 in term 0.
I20260927 15:04:57.083678 14761 leader_election.cc:302] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ca123d85e2145d4b952652d75416338, 36c453d1f2464b2993eb76e556f6178f, 8c51e814a0ef466abc5d7cd90d020d98; no voters:
I20260927 15:04:57.083953 14880 raft_consensus.cc:2781] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:04:57.084040 14880 raft_consensus.cc:482] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:04:57.084084 14880 raft_consensus.cc:3037] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:57.085095 14880 raft_consensus.cc:504] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.085415 14880 leader_election.cc:288] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 1 election: Requested vote from peers 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419), 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
I20260927 15:04:57.085753 14966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81633a49552143fa83d864c0b69a7f78" candidate_uuid: "8c51e814a0ef466abc5d7cd90d020d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca123d85e2145d4b952652d75416338"
I20260927 15:04:57.085836 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81633a49552143fa83d864c0b69a7f78" candidate_uuid: "8c51e814a0ef466abc5d7cd90d020d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c453d1f2464b2993eb76e556f6178f"
I20260927 15:04:57.085933 15098 raft_consensus.cc:3037] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:57.085963 14966 raft_consensus.cc:3037] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:04:57.086959 14966 raft_consensus.cc:2454] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c51e814a0ef466abc5d7cd90d020d98 in term 1.
I20260927 15:04:57.086992 15098 raft_consensus.cc:2454] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c51e814a0ef466abc5d7cd90d020d98 in term 1.
I20260927 15:04:57.087285 14763 leader_election.cc:302] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca123d85e2145d4b952652d75416338, 8c51e814a0ef466abc5d7cd90d020d98; no voters:
I20260927 15:04:57.087451 14880 raft_consensus.cc:2781] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:04:57.087673 14880 raft_consensus.cc:686] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 LEADER]: Becoming Leader. State: Replica: 8c51e814a0ef466abc5d7cd90d020d98, State: Running, Role: LEADER
I20260927 15:04:57.087932 14880 consensus_queue.cc:237] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.090859 14686 catalog_manager.cc:5495] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8c51e814a0ef466abc5d7cd90d020d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } health_report { overall_health: HEALTHY } } }
W20260927 15:04:57.099557 14875 tablet.cc:2369] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:04:57.113165 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.109654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.0.98.1:42599)
0927 15:04:57.109829 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:57.109833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:57.109848 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:04:57.109974 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:57.109976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:57.110028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:57.110140 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:04:57.110393 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.110906 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.111730 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:57.111969 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:57.112079 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:57.112364 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:04:57.112779 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:04:57.112906 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:57.112965 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 15:04:57.113019 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:04:57.117208 15098 raft_consensus.cc:1261] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 1 FOLLOWER]: Refusing update from remote peer 8c51e814a0ef466abc5d7cd90d020d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:04:57.117208 14966 raft_consensus.cc:1261] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Refusing update from remote peer 8c51e814a0ef466abc5d7cd90d020d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:04:57.118048 14880 consensus_queue.cc:1035] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:57.118261 15164 consensus_queue.cc:1035] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:04:57.125151 15167 mvcc.cc:205] Tried to move back new op lower bound from 7333976052184883200 to 7333976052074696704. Current Snapshot: MvccSnapshot[applied={T|T < 7333976052184883200}]
I20260927 15:04:57.126978 15169 mvcc.cc:205] Tried to move back new op lower bound from 7333976052184883200 to 7333976052074696704. Current Snapshot: MvccSnapshot[applied={T|T < 7333976052184883200}]
I20260927 15:04:57.127586 15168 mvcc.cc:205] Tried to move back new op lower bound from 7333976052184883200 to 7333976052074696704. Current Snapshot: MvccSnapshot[applied={T|T < 7333976052184883200}]
I20260927 15:04:57.134066 14684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:04:57.134234 14684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:04:57.136564 14684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Sending DeleteTablet for 3 replicas of tablet 81633a49552143fa83d864c0b69a7f78
I20260927 15:04:57.137310 14946 tablet_service.cc:1514] Processing DeleteTablet for tablet 81633a49552143fa83d864c0b69a7f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35582
I20260927 15:04:57.137369 14808 tablet_service.cc:1514] Processing DeleteTablet for tablet 81633a49552143fa83d864c0b69a7f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43600
I20260927 15:04:57.137585 14883 tablet_replica.cc:330] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: stopping tablet replica
I20260927 15:04:57.137708 14883 raft_consensus.cc:2229] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:04:57.137821 15176 tablet_replica.cc:330] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: stopping tablet replica
I20260927 15:04:57.137806 15078 tablet_service.cc:1514] Processing DeleteTablet for tablet 81633a49552143fa83d864c0b69a7f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:04:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36472
I20260927 15:04:57.138010 14883 raft_consensus.cc:2258] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:04:57.138494 15177 tablet_replica.cc:330] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: stopping tablet replica
I20260927 15:04:57.138706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14724
I20260927 15:04:57.138778 14883 ts_tablet_manager.cc:1905] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:57.138120 15176 raft_consensus.cc:2229] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:04:57.140522 15176 raft_consensus.cc:2258] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:04:57.140829 15177 raft_consensus.cc:2229] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:04:57.141125 15177 raft_consensus.cc:2258] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:04:57.142119 15176 ts_tablet_manager.cc:1905] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:57.142527 15177 ts_tablet_manager.cc:1905] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:57.144794 15176 ts_tablet_manager.cc:1918] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:57.144833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14878
W20260927 15:04:57.144960 14670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42599 (error 108)
I20260927 15:04:57.146781 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.146425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42599 (local address 127.0.0.1:43610)
0927 15:04:57.146579 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.146672 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:04:57.146955 14670 catalog_manager.cc:4584] TS 8c51e814a0ef466abc5d7cd90d020d98 (127.0.98.1:42599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81633a49552143fa83d864c0b69a7f78: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42599: connect: Connection refused (error 111)
I20260927 15:04:57.150648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 15015
I20260927 15:04:57.150753 15177 ts_tablet_manager.cc:1918] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:57.151566 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.150902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33419 (local address 127.0.0.1:35590)
0927 15:04:57.151391 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.151459 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
I20260927 15:04:57.156235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 14653
I20260927 15:04:57.162763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--webserver_interface=127.0.98.62
--webserver_port=33739
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:04:57.271827 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:57.272092 15179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:57.272131 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:57.276238 15179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:04:57.276296 15179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:57.276316 15179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:57.276377 15179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:04:57.276405 15179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:04:57.281543 15179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:57.282564 15179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:57.283587 15179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:57.292135 15187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:57.292104 15184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:57.292228 15179 server_base.cc:1034] running on GCE node
W20260927 15:04:57.292124 15185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:57.293006 15179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:57.294171 15179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:57.295362 15179 hybrid_clock.cc:648] HybridClock initialized: now 1790521497295346 us; error 38 us; skew 500 ppm
Sep 27 15:04:57 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:57.301344 15179 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:04:57.302330 15179 webserver.cc:460] Webserver started at http://127.0.98.62:33739/ using document root <none> and password file <none>
I20260927 15:04:57.302897 15179 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:57.302990 15179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:57.306777 15179 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:04:57.308851 15194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:57.310096 15179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:04:57.310230 15179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "8c5bbfe33a8440e7aec3b580038da264"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "6df9383050e3a970ec60dc6fe157388b"
server_key_iv: "e2671032752fc70f5b9b965990747857"
server_key_version: "encryptionkey@0"
I20260927 15:04:57.310770 15179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:57.326849 15179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:57.329885 15179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:57.330080 15179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:57.337033 15179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34171
I20260927 15:04:57.337313 15246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34171 every 8 connection(s)
I20260927 15:04:57.338174 15179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:04:57.338997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15179
I20260927 15:04:57.339479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42599
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=37593
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:57.343626 15247 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:04:57.346187 15247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Bootstrap starting.
I20260927 15:04:57.369259 15247 log.cc:824] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:57.383301 15247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:04:57.383814 15247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264: Bootstrap complete.
I20260927 15:04:57.389351 15247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:57.389992 15247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5bbfe33a8440e7aec3b580038da264, State: Initialized, Role: FOLLOWER
I20260927 15:04:57.390580 15247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:57.390738 15247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:04:57.390822 15247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:04:57.390930 15247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:04:57.392976 15247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:57.393332 15247 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c5bbfe33a8440e7aec3b580038da264; no voters:
I20260927 15:04:57.393676 15247 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:04:57.393811 15251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:04:57.394058 15251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [term 3 LEADER]: Becoming Leader. State: Replica: 8c5bbfe33a8440e7aec3b580038da264, State: Running, Role: LEADER
I20260927 15:04:57.394371 15251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260927 15:04:57.395007 15247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:04:57.395385 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5bbfe33a8440e7aec3b580038da264. Latest consensus state: current_term: 3 leader_uuid: "8c5bbfe33a8440e7aec3b580038da264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260927 15:04:57.395499 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:57.395758 15252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8c5bbfe33a8440e7aec3b580038da264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5bbfe33a8440e7aec3b580038da264" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260927 15:04:57.395839 15255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:04:57.395869 15252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5bbfe33a8440e7aec3b580038da264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:04:57.401227 15255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:04:57.401882 15255 catalog_manager.cc:1259] Loaded cluster ID: 69a099b022cb4d2cb708026f4fafc19b
I20260927 15:04:57.401937 15255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 15:04:57.402884 15255 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1c72d139ad google::LogMessage::Fail() at ??:0
@ 0x7f1c72d1688a google::LogMessage::SendToLog() at ??:0
@ 0x7f1c72d134de google::LogMessage::Flush() at ??:0
@ 0x7f1c72d14a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1c7d652fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1c7d653d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1c7d650579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1c7d694640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1c74f653d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1c74f65c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1c74f673f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1c74f56512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1c7659a6db start_thread at ??:0
@ 0x7f1c7181161f clone at ??:0
W20260927 15:04:57.487495 15249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:57.487803 15249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:57.487874 15249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:57.492000 15249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:57.492084 15249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:57.492172 15249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:04:57.497840 15249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37593
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:57.498893 15249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:57.499912 15249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:57.509389 15289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:57.509413 15288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:57.509413 15291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:57.509821 15249 server_base.cc:1034] running on GCE node
I20260927 15:04:57.510222 15249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:57.510846 15249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:57.512033 15249 hybrid_clock.cc:648] HybridClock initialized: now 1790521497512011 us; error 40 us; skew 500 ppm
Sep 27 15:04:57 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:57.514770 15249 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:04:57.515594 15249 webserver.cc:460] Webserver started at http://127.0.98.1:37593/ using document root <none> and password file <none>
I20260927 15:04:57.516222 15249 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:57.516309 15249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:57.520224 15249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:04:57.522485 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:57.523767 15249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:04:57.523900 15249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "8c51e814a0ef466abc5d7cd90d020d98"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "632584e2de9e0539dc3175cd4e3b5de0"
server_key_iv: "ed6f239e4eef174fb9e229245a586fb6"
server_key_version: "encryptionkey@0"
I20260927 15:04:57.524437 15249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:57.537649 15249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:57.540959 15249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:57.541170 15249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:57.541814 15249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:57.542933 15305 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:04:57.548445 15249 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:04:57.548545 15249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 15:04:57.548631 15249 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:04:57.550508 15249 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:04:57.550581 15249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 15:04:57.550901 15305 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap starting.
I20260927 15:04:57.557756 15305 log.cc:824] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Log is configured to *not* fsync() on all Append() calls
I20260927 15:04:57.561547 15249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42599
I20260927 15:04:57.561746 15413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42599 every 8 connection(s)
I20260927 15:04:57.562492 15249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:04:57.566958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15249
I20260927 15:04:57.567369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33419
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=43153
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:57.567718 15305 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:04:57.568149 15305 tablet_bootstrap.cc:492] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Bootstrap complete.
I20260927 15:04:57.568509 15305 ts_tablet_manager.cc:1397] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260927 15:04:57.571681 15305 raft_consensus.cc:348] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.572263 15305 raft_consensus.cc:729] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51e814a0ef466abc5d7cd90d020d98, State: Initialized, Role: FOLLOWER
I20260927 15:04:57.572898 15305 consensus_queue.cc:260] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:57.574007 15305 ts_tablet_manager.cc:1428] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260927 15:04:57.638293 15416 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:04:57.564373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:59311)
0927 15:04:57.564855 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.564908 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:57.565743 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:57.637480 (+ 71737us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":143,"threads_started":1}
W20260927 15:04:57.638798 15414 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34171 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34171: Client connection negotiation failed: client connection to 127.0.98.62:34171: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:04:57.639535 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.639304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:46055)
0927 15:04:57.639424 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.639482 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:04:57.640205 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.640031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:41441)
0927 15:04:57.640122 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.640160 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:04:57.640439 15414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:04:57.685873 15417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:57.686126 15417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:57.686200 15417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:57.690191 15417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:57.690254 15417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:57.690351 15417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:04:57.695530 15417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43153
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:57.696553 15417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:57.697557 15417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:04:57.707299 15417 server_base.cc:1034] running on GCE node
W20260927 15:04:57.707264 15424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:57.707278 15423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:57.707264 15426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:57.707870 15417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:57.708492 15417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:57.709679 15417 hybrid_clock.cc:648] HybridClock initialized: now 1790521497709653 us; error 37 us; skew 500 ppm
Sep 27 15:04:57 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:57.712337 15417 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:04:57.713133 15417 webserver.cc:460] Webserver started at http://127.0.98.2:43153/ using document root <none> and password file <none>
I20260927 15:04:57.713658 15417 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:57.713717 15417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:57.717655 15417 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:04:57.719762 15433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:57.720963 15417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:04:57.721081 15417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "0ca123d85e2145d4b952652d75416338"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "2260fc4b261cf239104bf25f8dbae685"
server_key_iv: "3b44b4effd3bf211d37ea1d50051f589"
server_key_version: "encryptionkey@0"
I20260927 15:04:57.721642 15417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:57.760798 15417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:57.763815 15417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:57.764016 15417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:57.764605 15417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:57.765684 15440 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 15:04:57.769618 15440 ts_tablet_manager.cc:1852] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 15:04:57.770072 15440 ts_tablet_manager.cc:1905] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:57.774066 15440 ts_tablet_manager.cc:1918] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:57.774207 15440 log.cc:1196] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/81633a49552143fa83d864c0b69a7f78
I20260927 15:04:57.774585 15440 ts_tablet_manager.cc:1939] T 81633a49552143fa83d864c0b69a7f78 P 0ca123d85e2145d4b952652d75416338: Deleting consensus metadata
I20260927 15:04:57.774917 15417 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 15:04:57.774974 15417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260927 15:04:57.775048 15417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:57.775082 15417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:57.784685 15417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33419
I20260927 15:04:57.784732 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33419 every 8 connection(s)
I20260927 15:04:57.785584 15417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:04:57.787524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15417
I20260927 15:04:57.787972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39615
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=34449
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:04:57.789389 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.787817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:34389)
0927 15:04:57.788311 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.788450 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":128,"threads_started":1}
W20260927 15:04:57.789885 15548 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34171 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34171: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
I20260927 15:04:57.790583 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.790288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:42477)
0927 15:04:57.790431 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.790500 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:04:57.791333 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.791056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:37825)
0927 15:04:57.791189 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.791256 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:04:57.791500 15548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:04:57.901290 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:04:57.901559 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:04:57.901602 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:04:57.905685 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:04:57.905748 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:04:57.905824 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:04:57.910993 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34449
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:04:57.912061 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:04:57.913084 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:04:57.922802 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:57.922811 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:04:57.922811 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:04:57.923079 15551 server_base.cc:1034] running on GCE node
I20260927 15:04:57.923461 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:04:57.924082 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:04:57.925258 15551 hybrid_clock.cc:648] HybridClock initialized: now 1790521497925226 us; error 41 us; skew 500 ppm
Sep 27 15:04:57 dist-test-slave-ml7t krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:04:57.928130 15551 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:04:57.929092 15551 webserver.cc:460] Webserver started at http://127.0.98.3:34449/ using document root <none> and password file <none>
I20260927 15:04:57.929651 15551 fs_manager.cc:359] Metadata directory not provided
I20260927 15:04:57.929754 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:04:57.933488 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 15:04:57.935889 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:57.937161 15551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:04:57.937321 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "36c453d1f2464b2993eb76e556f6178f"
format_stamp: "Formatted at 2026-09-27 15:04:55 on dist-test-slave-ml7t"
server_key: "aefddf8f820caf5f7f123c508c762837"
server_key_iv: "d019e251c2ab06cf1d6ae5178726d5d0"
server_key_version: "encryptionkey@0"
I20260927 15:04:57.937958 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:04:57.956907 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:04:57.960215 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:04:57.960433 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:04:57.961063 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:04:57.962209 15573 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 15:04:57.966022 15573 ts_tablet_manager.cc:1852] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 15:04:57.966416 15573 ts_tablet_manager.cc:1905] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:04:57.970410 15573 ts_tablet_manager.cc:1918] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:04:57.970515 15573 log.cc:1196] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/81633a49552143fa83d864c0b69a7f78
I20260927 15:04:57.970884 15573 ts_tablet_manager.cc:1939] T 81633a49552143fa83d864c0b69a7f78 P 36c453d1f2464b2993eb76e556f6178f: Deleting consensus metadata
I20260927 15:04:57.971230 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 15:04:57.971341 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260927 15:04:57.971442 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:04:57.971498 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:04:57.982208 15551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39615
I20260927 15:04:57.982285 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39615 every 8 connection(s)
I20260927 15:04:57.983229 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:04:57.985262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15551
I20260927 15:04:57.987381 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.985703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:32949)
0927 15:04:57.986294 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.986463 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":147,"threads_started":1}
W20260927 15:04:57.987854 15681 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34171 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34171: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
I20260927 15:04:57.988435 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.988208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:60579)
0927 15:04:57.988328 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.988379 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:04:57.989207 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:57.988992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:35567)
0927 15:04:57.989114 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:57.989159 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 15:04:57.989431 15681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:04:58.641601 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:58.641155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:38783)
0927 15:04:58.641439 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:58.641504 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260927 15:04:58.792513 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:58.792104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:58685)
0927 15:04:58.792361 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:58.792426 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260927 15:04:58.990381 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:58.989988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:37471)
0927 15:04:58.990239 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:58.990296 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260927 15:04:59.008055 15688 raft_consensus.cc:482] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:04:59.008306 15688 raft_consensus.cc:504] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:04:59.009426 15688 leader_election.cc:288] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419), 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
Sep 27 15:04:59 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 15:04:59 dist-test-slave-ml7t krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 15:04:59.024538 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:59.009897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49029 (local address 127.0.98.3:39615)
0927 15:04:59.010467 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:59.010478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:59.010509 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:04:59.011160 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:59.011165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:59.011466 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:59.011787 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:59.013163 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.014197 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.015902 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.016379 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.016560 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:59.019669 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:59.019677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:59.019692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:59.019726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:59.021870 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:59.022465 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:59.022473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:59.022479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:59.022610 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:59.022951 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:59.022955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:59.022957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:59.023364 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:59.023515 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:59.024031 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 15:04:59.024306 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":95,"threads_started":1}
I20260927 15:04:59.024539 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:59.010021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53425 (local address 127.0.98.2:33419)
0927 15:04:59.010348 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 15:04:59.010360 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:04:59.010395 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:04:59.011126 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:04:59.011133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:04:59.011466 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:04:59.011786 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:04:59.013201 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.014439 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.015934 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.016379 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.016553 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:59.019438 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:04:59.019444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:04:59.019457 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:04:59.019486 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:04:59.021870 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:59.022496 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:59.022501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:59.022505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:59.022624 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:04:59.022982 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:04:59.022985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:04:59.022986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:04:59.023359 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:04:59.023496 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:04:59.024018 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 15:04:59.024296 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260927 15:04:59.024538 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:59.009737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33419 (local address 127.0.98.1:53425)
0927 15:04:59.010173 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:59.010188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:59.010290 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:59.011832 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:59.011838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:59.012406 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:59.012972 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:59.012988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.014601 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.014605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:59.015778 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:59.015788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.016507 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.016510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:59.016718 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:59.017379 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:59.017395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:59.019123 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:59.022045 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:59.022048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:59.022049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:59.022352 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:59.022747 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:59.022749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:59.022751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:59.022859 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:59.023483 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:59.023490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:59.023715 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 15:04:59.023904 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 15:04:59.024243 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":292,"spinlock_wait_cycles":17536,"thread_start_us":77,"threads_started":1}
I20260927 15:04:59.024557 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:59.009737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39615 (local address 127.0.98.1:49029)
0927 15:04:59.009947 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:59.009964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:04:59.010090 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:04:59.011833 (+ 1743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:04:59.011839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:04:59.012407 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:04:59.012973 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:59.012988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.014388 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.014392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:59.015771 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:04:59.015781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:04:59.016509 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:04:59.016512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:04:59.016707 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:04:59.017442 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:04:59.017454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:04:59.019453 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:04:59.021991 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:59.021996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:59.022008 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:59.022346 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:59.022712 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:04:59.022716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:04:59.022718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:04:59.022853 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:04:59.023497 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:04:59.023501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:04:59.023719 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 15:04:59.023903 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 15:04:59.024327 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":237,"spinlock_wait_cycles":25472}
W20260927 15:04:59.027303 15300 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
W20260927 15:04:59.027549 15302 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
I20260927 15:04:59.027654 15302 leader_election.cc:302] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c51e814a0ef466abc5d7cd90d020d98; no voters: 0ca123d85e2145d4b952652d75416338, 36c453d1f2464b2993eb76e556f6178f
I20260927 15:04:59.027956 15688 raft_consensus.cc:2726] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:04:59.642910 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:59.642468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:35817)
0927 15:04:59.642741 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:59.642799 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":71,"threads_started":1}
I20260927 15:04:59.793759 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:59.793305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:35087)
0927 15:04:59.793601 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:59.793660 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260927 15:04:59.991568 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:04:59.991143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:46323)
0927 15:04:59.991403 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:04:59.991465 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
I20260927 15:05:00.644137 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:00.643680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:53521)
0927 15:05:00.643974 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:00.644034 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
I20260927 15:05:00.794948 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:00.794509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:39339)
0927 15:05:00.794790 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:00.794855 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":67,"threads_started":1}
I20260927 15:05:00.992786 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:00.992297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:47047)
0927 15:05:00.992605 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:00.992685 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260927 15:05:01.123510 15698 raft_consensus.cc:482] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:05:01.123631 15698 raft_consensus.cc:504] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:05:01.124111 15698 leader_election.cc:288] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419), 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
W20260927 15:05:01.124972 15300 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
W20260927 15:05:01.125059 15302 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
I20260927 15:05:01.125177 15302 leader_election.cc:302] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c51e814a0ef466abc5d7cd90d020d98; no voters: 0ca123d85e2145d4b952652d75416338, 36c453d1f2464b2993eb76e556f6178f
I20260927 15:05:01.125310 15698 raft_consensus.cc:2726] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:05:01.645372 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:01.644907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:33715)
0927 15:05:01.645206 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:01.645263 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260927 15:05:01.796203 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:01.795787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:51411)
0927 15:05:01.796032 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:01.796086 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260927 15:05:01.993983 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:01.993510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:52843)
0927 15:05:01.993763 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:01.993882 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
I20260927 15:05:02.646986 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:02.646430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:38041)
0927 15:05:02.646729 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:02.646791 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":73,"threads_started":1}
I20260927 15:05:02.797557 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:02.797109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:44235)
0927 15:05:02.797388 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:02.797457 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260927 15:05:02.995194 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:02.994762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:57597)
0927 15:05:02.995037 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:02.995098 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260927 15:05:03.648295 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:03.647880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:50221)
0927 15:05:03.648138 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:03.648200 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260927 15:05:03.798763 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:03.798364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:45119)
0927 15:05:03.798595 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:03.798648 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260927 15:05:03.996451 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:03.996020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:56703)
0927 15:05:03.996289 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:03.996350 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260927 15:05:04.083783 15708 raft_consensus.cc:482] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:05:04.083920 15708 raft_consensus.cc:504] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:05:04.084429 15708 leader_election.cc:288] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419), 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
W20260927 15:05:04.085213 15302 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
W20260927 15:05:04.085294 15300 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
I20260927 15:05:04.085366 15300 leader_election.cc:302] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c51e814a0ef466abc5d7cd90d020d98; no voters: 0ca123d85e2145d4b952652d75416338, 36c453d1f2464b2993eb76e556f6178f
I20260927 15:05:04.085522 15708 raft_consensus.cc:2726] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:05:04.649663 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:04.649146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:39555)
0927 15:05:04.649481 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:04.649562 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260927 15:05:04.800037 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:04.799549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:45885)
0927 15:05:04.799880 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:04.799942 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260927 15:05:04.997826 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:04.997215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:53717)
0927 15:05:04.997589 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:04.997662 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":89,"threads_started":1}
I20260927 15:05:05.650923 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:05.650439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:47167)
0927 15:05:05.650763 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:05.650829 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260927 15:05:05.801156 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:05.800778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:57569)
0927 15:05:05.801009 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:05.801062 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260927 15:05:05.999182 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:05.998718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:58527)
0927 15:05:05.999017 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:05.999082 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260927 15:05:06.652104 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:06.651702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:46557)
0927 15:05:06.651955 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:06.652012 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260927 15:05:06.802451 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:06.801975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:42857)
0927 15:05:06.802276 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:06.802345 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20260927 15:05:07.000356 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:06.999959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:49783)
0927 15:05:07.000211 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:07.000267 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260927 15:05:07.653445 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:07.653020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:48705)
0927 15:05:07.653295 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:07.653353 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
I20260927 15:05:07.803911 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:07.803463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:58529)
0927 15:05:07.803753 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:07.803815 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260927 15:05:08.001653 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:08.001213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:35793)
0927 15:05:08.001481 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:08.001555 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260927 15:05:08.654651 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:08.654248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:60109)
0927 15:05:08.654498 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:08.654555 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260927 15:05:08.805034 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:08.804641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:35105)
0927 15:05:08.804888 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:08.804940 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260927 15:05:08.894598 15723 raft_consensus.cc:482] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:05:08.894731 15723 raft_consensus.cc:504] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ca123d85e2145d4b952652d75416338" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33419 } } peers { permanent_uuid: "36c453d1f2464b2993eb76e556f6178f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39615 } } peers { permanent_uuid: "8c51e814a0ef466abc5d7cd90d020d98" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42599 } }
I20260927 15:05:08.895190 15723 leader_election.cc:288] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419), 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615)
W20260927 15:05:08.896005 15302 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ca123d85e2145d4b952652d75416338 (127.0.98.2:33419): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
W20260927 15:05:08.896106 15300 leader_election.cc:341] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36c453d1f2464b2993eb76e556f6178f (127.0.98.3:39615): Not found: Tablet not found: 81633a49552143fa83d864c0b69a7f78
I20260927 15:05:08.896178 15300 leader_election.cc:302] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c51e814a0ef466abc5d7cd90d020d98; no voters: 0ca123d85e2145d4b952652d75416338, 36c453d1f2464b2993eb76e556f6178f
I20260927 15:05:08.896313 15723 raft_consensus.cc:2726] T 81633a49552143fa83d864c0b69a7f78 P 8c51e814a0ef466abc5d7cd90d020d98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:05:09.002962 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:09.002499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:41391)
0927 15:05:09.002804 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:09.002866 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
I20260927 15:05:09.655869 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:09.655439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:60793)
0927 15:05:09.655714 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:09.655775 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260927 15:05:09.806200 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:09.805736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:60075)
0927 15:05:09.806029 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:09.806080 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260927 15:05:10.004231 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:10.003799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:51801)
0927 15:05:10.004068 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:10.004127 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260927 15:05:10.657138 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:10.656643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:36245)
0927 15:05:10.656915 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:10.657017 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260927 15:05:10.807317 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:10.806903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:33491)
0927 15:05:10.807152 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:10.807217 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
I20260927 15:05:11.005585 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:11.005137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:49977)
0927 15:05:11.005420 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:11.005485 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260927 15:05:11.658450 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:11.658041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:36609)
0927 15:05:11.658289 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:11.658352 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260927 15:05:11.808794 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:11.808235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:40223)
0927 15:05:11.808578 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:11.808658 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":122,"threads_started":1}
I20260927 15:05:12.006887 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:12.006390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:44817)
0927 15:05:12.006726 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:12.006790 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":90,"threads_started":1}
I20260927 15:05:12.659811 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:12.659371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:47329)
0927 15:05:12.659651 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:12.659710 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":70,"threads_started":1}
I20260927 15:05:12.810223 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:12.809677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:55205)
0927 15:05:12.810067 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:12.810126 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":101,"threads_started":1}
I20260927 15:05:12.996681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 15249
I20260927 15:05:13.002731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 15417
I20260927 15:05:13.008395 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:13.007936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:53985)
0927 15:05:13.008238 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:13.008302 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260927 15:05:13.008857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 15551
2026-09-27T15:05:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19157 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:05:13 dist-test-slave-ml7t krb5kdc[15739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:05:13 dist-test-slave-ml7t krb5kdc[15739](info): set up 2 sockets
Sep 27 15:05:13 dist-test-slave-ml7t krb5kdc[15739](info): commencing operation
krb5kdc: starting...
W20260927 15:05:14.046514 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:05:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:05:14.166683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:05:14.273088 15752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:14.273346 15752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:14.273384 15752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:14.277318 15752 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:05:14.277383 15752 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:05:14.277400 15752 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:05:14.277413 15752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:05:14.277431 15752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:14.277449 15752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:14.277465 15752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:05:14.277480 15752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:05:14.282675 15752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:14.283654 15752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:14.284592 15752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:14.293152 15757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:14.293238 15752 server_base.cc:1034] running on GCE node
W20260927 15:05:14.293313 15758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:14.293433 15760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:14.294116 15752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:14.295161 15752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:14.296321 15752 hybrid_clock.cc:648] HybridClock initialized: now 1790521514296292 us; error 39 us; skew 500 ppm
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:14.299165 15752 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:05:14.299930 15752 webserver.cc:460] Webserver started at http://127.0.98.62:42103/ using document root <none> and password file <none>
I20260927 15:05:14.300438 15752 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:14.300493 15752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:14.300647 15752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:05:14.302599 15752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "278949155b6040eaa593d71530085b11"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "14ec7c63f08a579ff03469cb31330215"
server_key_iv: "507e2ee9cd8612ef00d14200d5242355"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.303090 15752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "278949155b6040eaa593d71530085b11"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "14ec7c63f08a579ff03469cb31330215"
server_key_iv: "507e2ee9cd8612ef00d14200d5242355"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.306684 15752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:05:14.309058 15767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:14.310477 15752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:05:14.310580 15752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "278949155b6040eaa593d71530085b11"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "14ec7c63f08a579ff03469cb31330215"
server_key_iv: "507e2ee9cd8612ef00d14200d5242355"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.310673 15752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:14.328265 15752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:14.331262 15752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:14.331478 15752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:14.338430 15752 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33065
I20260927 15:05:14.338752 15819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33065 every 8 connection(s)
I20260927 15:05:14.339408 15752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:05:14.342106 15820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:05:14.342598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15752
I20260927 15:05:14.342743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:05:14.343030 392 external_mini_cluster.cc:1428] Setting key 3ec65649daa07db5da1e43e11b19283f
I20260927 15:05:14.348901 15820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Bootstrap starting.
I20260927 15:05:14.351557 15820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Neither blocks nor log segments found. Creating new log.
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:05:14.352365 15820 log.cc:824] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:14.354669 15820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: No bootstrap required, opened a new log
I20260927 15:05:14.357645 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:14.344323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.0.98.62:33065)
0927 15:05:14.344702 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:14.344722 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:14.344765 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:05:14.345447 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:14.345464 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:14.345829 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:14.346112 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:14.346995 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.347953 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.349095 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.349540 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.349700 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:14.352469 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:14.352474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:14.352511 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:14.352543 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:14.354547 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:14.354994 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:14.355001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:14.355009 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:14.355147 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:14.355415 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:14.355418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:14.355420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:14.355910 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:14.356090 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:14.356298 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 15:05:14.356588 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":133,"threads_started":1}
I20260927 15:05:14.359243 15820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:14.359452 15820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:05:14.359516 15820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278949155b6040eaa593d71530085b11, State: Initialized, Role: FOLLOWER
I20260927 15:05:14.359957 15820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:14.360132 15820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:05:14.360198 15820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:05:14.360294 15820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:05:14.361182 15820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:14.361524 15820 leader_election.cc:302] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278949155b6040eaa593d71530085b11; no voters:
I20260927 15:05:14.361913 15820 leader_election.cc:288] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:05:14.361990 15825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:05:14.362282 15825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 LEADER]: Becoming Leader. State: Replica: 278949155b6040eaa593d71530085b11, State: Running, Role: LEADER
I20260927 15:05:14.362624 15825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:14.363132 15820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:05:14.364274 15827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278949155b6040eaa593d71530085b11. Latest consensus state: current_term: 1 leader_uuid: "278949155b6040eaa593d71530085b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260927 15:05:14.364405 15827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: This master's current role is: LEADER
I20260927 15:05:14.364720 15826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278949155b6040eaa593d71530085b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260927 15:05:14.364805 15834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:05:14.364814 15826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: This master's current role is: LEADER
I20260927 15:05:14.367895 15834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:05:14.374037 15834 catalog_manager.cc:1347] Generated new cluster ID: c2e0bf845ac643f6b73ad1ea1eecd565
I20260927 15:05:14.374104 15834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:05:14.387669 15834 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:05:14.388710 15834 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:05:14.399470 15834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Generated new TSK 0
I20260927 15:05:14.400172 15834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:05:14.433341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:05:14.545310 15846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:14.545565 15846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:14.545607 15846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:14.549731 15846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:14.549799 15846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:14.549888 15846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:05:14.554982 15846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:14.555999 15846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:14.557006 15846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:14.566530 15852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:14.566532 15851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:14.566629 15854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:14.566715 15846 server_base.cc:1034] running on GCE node
I20260927 15:05:14.567147 15846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:14.567766 15846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:14.568957 15846 hybrid_clock.cc:648] HybridClock initialized: now 1790521514568943 us; error 27 us; skew 500 ppm
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:14.571672 15846 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:05:14.572564 15846 webserver.cc:460] Webserver started at http://127.0.98.1:46867/ using document root <none> and password file <none>
I20260927 15:05:14.573107 15846 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:14.573201 15846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:14.573419 15846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:05:14.575471 15846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "cbcfd79ba9eb489990091414aca5d91c"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "5748cb29ace45d296111cf057af7cd30"
server_key_iv: "eafdd0df4a623d743a60301556b4222d"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.576035 15846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cbcfd79ba9eb489990091414aca5d91c"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "5748cb29ace45d296111cf057af7cd30"
server_key_iv: "eafdd0df4a623d743a60301556b4222d"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.579787 15846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:05:14.582110 15861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:14.583336 15846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:05:14.583463 15846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "cbcfd79ba9eb489990091414aca5d91c"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "5748cb29ace45d296111cf057af7cd30"
server_key_iv: "eafdd0df4a623d743a60301556b4222d"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.583575 15846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:14.598981 15846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:14.601649 15846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:14.601877 15846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:14.602433 15846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:14.603224 15846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:05:14.603291 15846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:14.603360 15846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:05:14.603401 15846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:14.612380 15846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37025
I20260927 15:05:14.612403 15974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37025 every 8 connection(s)
I20260927 15:05:14.613451 15846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:05:14.619648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15846
I20260927 15:05:14.619800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:05:14.620146 392 external_mini_cluster.cc:1428] Setting key 7d62e10386ce77034b3be52f50dde71a
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:05:14.627532 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:14.615625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42219 (local address 127.0.98.62:33065)
0927 15:05:14.615877 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:14.615882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:14.616692 (+ 810us) server_negotiation.cc:407] Connection header received
0927 15:05:14.618006 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:14.618009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:14.618055 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:14.618145 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:14.619290 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.619884 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.621114 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.621365 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.621478 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:14.623931 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:14.623940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:14.623942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:14.623965 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:14.625412 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:14.626028 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:14.626032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:14.626034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:14.626114 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:14.626457 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:14.626460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:14.626462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:14.626648 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:14.626751 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:14.627241 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 15:05:14.627366 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260927 15:05:14.628353 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:14.616079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:42219)
0927 15:05:14.616537 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:14.616583 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:14.617740 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:14.618307 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:14.618315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:14.618683 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:14.619165 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:14.619178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.620054 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.620056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:14.620986 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:14.620997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.621517 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.621520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:14.621692 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:14.622349 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:14.622366 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:14.623770 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:14.625569 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:14.625575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:14.625590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:14.625927 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:14.626213 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:14.626217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:14.626220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:14.626358 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:14.626751 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:14.626759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:14.627002 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 15:05:14.627187 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 15:05:14.627516 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:05:14.629618 15975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260927 15:05:14.629913 15975 heartbeater.cc:464] Registering TS with master...
I20260927 15:05:14.630409 15975 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:05:14.632071 15784 ts_manager.cc:194] Registered new tserver with Master: cbcfd79ba9eb489990091414aca5d91c (127.0.98.1:37025)
I20260927 15:05:14.633189 15784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:05:14.648685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:05:14.759436 15980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:14.759686 15980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:14.759729 15980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:14.763834 15980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:14.763932 15980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:14.764072 15980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:05:14.769554 15980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:14.770597 15980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:14.771587 15980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:14.780861 15986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:14.780849 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:14.780877 15988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:14.781180 15980 server_base.cc:1034] running on GCE node
I20260927 15:05:14.781596 15980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:14.782222 15980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:14.783386 15980 hybrid_clock.cc:648] HybridClock initialized: now 1790521514783360 us; error 39 us; skew 500 ppm
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:14.786240 15980 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:05:14.787096 15980 webserver.cc:460] Webserver started at http://127.0.98.2:41249/ using document root <none> and password file <none>
I20260927 15:05:14.787647 15980 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:14.787709 15980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:14.787907 15980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:05:14.789975 15980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "c712fcc4fe7740099001b5baffdf132b"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cbcbc05e4b1b0a8f52740e719266ccf8"
server_key_iv: "56340a34ae8dbfdc938fb3ef12bedc5e"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.790532 15980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c712fcc4fe7740099001b5baffdf132b"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cbcbc05e4b1b0a8f52740e719266ccf8"
server_key_iv: "56340a34ae8dbfdc938fb3ef12bedc5e"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.794097 15980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:05:14.796535 15995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:14.797755 15980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:05:14.797930 15980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "c712fcc4fe7740099001b5baffdf132b"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cbcbc05e4b1b0a8f52740e719266ccf8"
server_key_iv: "56340a34ae8dbfdc938fb3ef12bedc5e"
server_key_version: "encryptionkey@0"
I20260927 15:05:14.798036 15980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:14.808696 15980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:14.811796 15980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:14.811998 15980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:14.812599 15980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:14.813401 15980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:05:14.813470 15980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:14.813539 15980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:05:14.813608 15980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:14.822651 15980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45343
I20260927 15:05:14.822691 16108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45343 every 8 connection(s)
I20260927 15:05:14.823519 15980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:05:14.825044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 15980
I20260927 15:05:14.825186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:05:14.825491 392 external_mini_cluster.cc:1428] Setting key e1e1ea74613120a5785e245bb84ce6d2
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:05:14.838366 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:14.825370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33125 (local address 127.0.98.62:33065)
0927 15:05:14.825502 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:14.825507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:14.826895 (+ 1388us) server_negotiation.cc:407] Connection header received
0927 15:05:14.827774 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:14.827778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:14.827828 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:14.827905 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:14.829230 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.829933 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.831125 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.831370 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.831513 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:14.834340 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:14.834344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:14.834351 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:14.834382 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:14.835983 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:14.836540 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:14.836544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:14.836545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:14.836661 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:14.837048 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:14.837051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:14.837053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:14.837338 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:14.837442 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:14.838113 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 15:05:14.838241 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:05:14.839497 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:14.825902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:33125)
0927 15:05:14.826757 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:14.826812 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:14.827573 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:14.828065 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:14.828071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:14.828412 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:14.828937 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:14.828949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.830086 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.830089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:14.830985 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:14.830994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:14.831560 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:14.831565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:14.831738 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:14.832416 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:14.832442 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:14.834166 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:14.836122 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:14.836128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:14.836142 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:14.836443 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:14.836762 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:14.836765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:14.836767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:14.836907 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:14.837466 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:14.837471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:14.837844 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 15:05:14.838075 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 15:05:14.838450 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":143,"threads_started":1}
I20260927 15:05:14.840739 16109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260927 15:05:14.840996 16109 heartbeater.cc:464] Registering TS with master...
I20260927 15:05:14.841629 16109 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260927 15:05:14.842808 15784 ts_manager.cc:194] Registered new tserver with Master: c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343)
I20260927 15:05:14.843484 15784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:05:14.855017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:05:14.969084 16114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:14.969367 16114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:14.969447 16114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:14.973501 16114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:14.973580 16114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:14.973687 16114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:05:14.979162 16114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:14.980175 16114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:14.981160 16114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:14.990700 16119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:14.990732 16120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:14.990809 16122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:14.991052 16114 server_base.cc:1034] running on GCE node
I20260927 15:05:14.991479 16114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:14.992117 16114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:14.993279 16114 hybrid_clock.cc:648] HybridClock initialized: now 1790521514993266 us; error 28 us; skew 500 ppm
Sep 27 15:05:14 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:14.995993 16114 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:05:14.996881 16114 webserver.cc:460] Webserver started at http://127.0.98.3:39087/ using document root <none> and password file <none>
I20260927 15:05:14.997429 16114 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:14.997521 16114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:14.997824 16114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:05:14.999822 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "43743b3a4f20406085872fdfa9025df8"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cc1a7276a1f459e810e5a94084aae72e"
server_key_iv: "80a2a88625d2d2c2d9f8967cf09b68b5"
server_key_version: "encryptionkey@0"
I20260927 15:05:15.000383 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "43743b3a4f20406085872fdfa9025df8"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cc1a7276a1f459e810e5a94084aae72e"
server_key_iv: "80a2a88625d2d2c2d9f8967cf09b68b5"
server_key_version: "encryptionkey@0"
I20260927 15:05:15.004307 16114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:05:15.007258 16129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:15.008503 16114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:05:15.008662 16114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "43743b3a4f20406085872fdfa9025df8"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cc1a7276a1f459e810e5a94084aae72e"
server_key_iv: "80a2a88625d2d2c2d9f8967cf09b68b5"
server_key_version: "encryptionkey@0"
I20260927 15:05:15.008788 16114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:15.030421 16114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:15.033361 16114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:15.033571 16114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:15.034220 16114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:15.035074 16114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:05:15.035149 16114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:15.035224 16114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:05:15.035267 16114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:15.044607 16114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42015
I20260927 15:05:15.044631 16242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42015 every 8 connection(s)
I20260927 15:05:15.045568 16114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:05:15.051687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 16114
I20260927 15:05:15.051816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:05:15.052105 392 external_mini_cluster.cc:1428] Setting key e630585c8bde73c23acf836aae80cd04
Sep 27 15:05:15 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:05:15.058856 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.047469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39395 (local address 127.0.98.62:33065)
0927 15:05:15.047608 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.047612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.048621 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 15:05:15.049480 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.049483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.049525 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.049598 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:15.050815 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.051341 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.052879 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.053172 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.053284 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.055779 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:15.055782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:15.055784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:15.055807 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:15.057049 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:15.057502 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:15.057504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:15.057505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:15.057578 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:15.057932 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:15.057935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:15.057936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:15.058105 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:15.058194 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.058640 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.058737 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:05:15.059687 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.047870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:39395)
0927 15:05:15.048492 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:15.048533 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:15.049302 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:15.049826 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:15.049841 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:15.050201 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:15.050687 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.050699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.051478 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.051484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.052742 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.052775 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.053323 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.053328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.053497 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.054187 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:15.054213 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:15.055634 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:15.057154 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:15.057159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:15.057170 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:15.057418 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:15.057660 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:15.057662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:15.057663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:15.057853 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:15.058201 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:15.058206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:15.058432 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 15:05:15.058580 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 15:05:15.058850 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":109,"threads_started":1}
I20260927 15:05:15.060807 16243 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260927 15:05:15.061051 16243 heartbeater.cc:464] Registering TS with master...
I20260927 15:05:15.061529 16243 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260927 15:05:15.062593 15784 ts_manager.cc:194] Registered new tserver with Master: 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015)
I20260927 15:05:15.063194 15784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39395
I20260927 15:05:15.065963 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:05:15.074604 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.067824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.98.62:33065)
0927 15:05:15.067943 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.067946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.068068 (+ 122us) server_negotiation.cc:407] Connection header received
0927 15:05:15.068164 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.068166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.068200 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.068264 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:15.068903 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.069386 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.070324 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.070530 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.070608 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.071805 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:15.071808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:15.071810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:15.071831 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:15.073072 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:15.073474 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:15.073477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:15.073479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:15.073602 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:15.073893 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:15.073896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:15.073897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:15.074112 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:15.074270 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.074397 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.074486 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:05:15.077620 15784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:05:15.080729 15784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:05:15.095399 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.089927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.98.2:45343)
0927 15:05:15.090598 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.090609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.090655 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:05:15.090729 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.090734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.090930 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.091083 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:15.091426 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.092579 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.094045 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.094707 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.094833 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.094983 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.095141 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.095237 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":138,"threads_started":1}
I20260927 15:05:15.095388 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.089474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45343 (local address 127.0.0.1:46674)
0927 15:05:15.090329 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:15.090343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:15.090423 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:15.091242 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:15.091245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:15.091260 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:15.091332 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.091338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.092767 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.092770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.093921 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.093930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.094830 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.094834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.094933 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.094969 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:05:15.095110 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 15:05:15.095245 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":47,"thread_start_us":75,"threads_started":1}
I20260927 15:05:15.095715 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.089927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.0.98.1:37025)
0927 15:05:15.090405 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.090420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.090442 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:05:15.090732 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.090736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.090992 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.091418 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:15.091566 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.092294 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.093997 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.094768 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.094877 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.095015 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.095450 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.095566 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":106,"threads_started":1}
I20260927 15:05:15.095696 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.089083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42015 (local address 127.0.0.1:40120)
0927 15:05:15.089438 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:15.089484 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:15.089603 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:15.091065 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:15.091072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:15.091094 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:15.091250 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.091257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.092724 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.092727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.094169 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.094178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.095354 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.095358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.095438 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.095459 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:05:15.095523 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:05:15.095578 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":45,"thread_start_us":115,"threads_started":1}
I20260927 15:05:15.095914 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.089182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:38204)
0927 15:05:15.089966 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:15.089981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:15.090298 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:15.091151 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:15.091153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:15.091171 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:15.091276 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.091282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.092424 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.092429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.093875 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.093885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.095082 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.095088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.095182 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.095204 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:05:15.095269 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:05:15.095343 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":22,"thread_start_us":92,"threads_started":1}
I20260927 15:05:15.096314 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.089942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.0.98.3:42015)
0927 15:05:15.090562 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.090572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.090588 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:05:15.090678 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.090686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.090912 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.091076 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:15.091362 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.092588 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.094284 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.095059 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.095170 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.095301 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.096065 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.096178 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":104,"threads_started":1}
I20260927 15:05:15.098678 15909 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac52a6eaa84a38b7402520b367dd7e (DEFAULT_TABLE table=test-table [id=926003d489274fc5a009c57aa34bc559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:05:15.098747 16043 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac52a6eaa84a38b7402520b367dd7e (DEFAULT_TABLE table=test-table [id=926003d489274fc5a009c57aa34bc559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:05:15.098748 16177 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac52a6eaa84a38b7402520b367dd7e (DEFAULT_TABLE table=test-table [id=926003d489274fc5a009c57aa34bc559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:05:15.099661 15909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac52a6eaa84a38b7402520b367dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:05:15.099723 16043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac52a6eaa84a38b7402520b367dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:05:15.099731 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac52a6eaa84a38b7402520b367dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:05:15.105871 16260 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Bootstrap starting.
I20260927 15:05:15.107429 16261 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Bootstrap starting.
I20260927 15:05:15.108009 16260 tablet_bootstrap.cc:654] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Neither blocks nor log segments found. Creating new log.
I20260927 15:05:15.109033 16260 log.cc:824] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:15.109495 16261 tablet_bootstrap.cc:654] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Neither blocks nor log segments found. Creating new log.
I20260927 15:05:15.109673 16262 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Bootstrap starting.
I20260927 15:05:15.110380 16261 log.cc:824] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:15.111115 16260 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: No bootstrap required, opened a new log
I20260927 15:05:15.111318 16260 ts_tablet_manager.cc:1397] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:05:15.111675 16262 tablet_bootstrap.cc:654] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Neither blocks nor log segments found. Creating new log.
I20260927 15:05:15.112269 16261 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: No bootstrap required, opened a new log
I20260927 15:05:15.112447 16261 ts_tablet_manager.cc:1397] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:05:15.112763 16262 log.cc:824] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:15.114566 16262 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: No bootstrap required, opened a new log
I20260927 15:05:15.114768 16262 ts_tablet_manager.cc:1397] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:05:15.115348 16260 raft_consensus.cc:348] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.115619 16260 raft_consensus.cc:374] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:05:15.115705 16260 raft_consensus.cc:729] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbcfd79ba9eb489990091414aca5d91c, State: Initialized, Role: FOLLOWER
I20260927 15:05:15.116158 16260 consensus_queue.cc:260] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.116165 16261 raft_consensus.cc:348] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.116438 16261 raft_consensus.cc:374] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:05:15.116549 16261 raft_consensus.cc:729] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c712fcc4fe7740099001b5baffdf132b, State: Initialized, Role: FOLLOWER
I20260927 15:05:15.116812 15975 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260927 15:05:15.117154 16260 ts_tablet_manager.cc:1428] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:05:15.118083 16261 consensus_queue.cc:260] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.118278 16262 raft_consensus.cc:348] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.118557 16262 raft_consensus.cc:374] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:05:15.118662 16262 raft_consensus.cc:729] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43743b3a4f20406085872fdfa9025df8, State: Initialized, Role: FOLLOWER
I20260927 15:05:15.118661 16109 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260927 15:05:15.118975 16261 ts_tablet_manager.cc:1428] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:05:15.119145 16262 consensus_queue.cc:260] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.119805 16243 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260927 15:05:15.119987 16262 ts_tablet_manager.cc:1428] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 15:05:15.296578 16244 tablet.cc:2369] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:05:15.324589 16110 tablet.cc:2369] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:05:15.364902 15976 tablet.cc:2369] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:05:15.366804 16266 raft_consensus.cc:482] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:05:15.366993 16266 raft_consensus.cc:504] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.368077 16266 leader_election.cc:288] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015), c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343)
I20260927 15:05:15.372238 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.368587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40817 (local address 127.0.98.2:45343)
0927 15:05:15.368731 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.368736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.368925 (+ 189us) server_negotiation.cc:407] Connection header received
0927 15:05:15.369081 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.369085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.369136 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.369227 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:15.369516 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.370077 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.371279 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.371670 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.371787 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.371851 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.372029 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.372099 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:05:15.372262 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.368384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45343 (local address 127.0.98.1:40817)
0927 15:05:15.368812 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:15.368824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:15.368964 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:15.369233 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:15.369236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:15.369251 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:15.369403 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.369408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.370189 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.370195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.371151 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.371160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.371792 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.371796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.371893 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.371916 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:05:15.372014 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:05:15.372092 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":95,"spinlock_wait_cycles":5376,"thread_start_us":122,"threads_started":1}
I20260927 15:05:15.372401 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.368505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56897 (local address 127.0.98.3:42015)
0927 15:05:15.368643 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.368648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.368892 (+ 244us) server_negotiation.cc:407] Connection header received
0927 15:05:15.369073 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.369077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.369137 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.369229 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:15.369542 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.370181 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.371341 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.371801 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.371929 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.371993 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.372185 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.372279 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:05:15.372424 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.368384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42015 (local address 127.0.98.1:56897)
0927 15:05:15.368767 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:15.368793 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:15.368938 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:15.369241 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:15.369244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:15.369260 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:15.369413 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.369419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.370298 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.370305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.371204 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:15.371213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.371934 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.371938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:15.372035 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.372066 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:05:15.372167 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 15:05:15.372262 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260927 15:05:15.373558 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac52a6eaa84a38b7402520b367dd7e" candidate_uuid: "cbcfd79ba9eb489990091414aca5d91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c712fcc4fe7740099001b5baffdf132b" is_pre_election: true
I20260927 15:05:15.373703 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac52a6eaa84a38b7402520b367dd7e" candidate_uuid: "cbcfd79ba9eb489990091414aca5d91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43743b3a4f20406085872fdfa9025df8" is_pre_election: true
I20260927 15:05:15.373926 16063 raft_consensus.cc:2454] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbcfd79ba9eb489990091414aca5d91c in term 0.
I20260927 15:05:15.374051 16197 raft_consensus.cc:2454] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbcfd79ba9eb489990091414aca5d91c in term 0.
I20260927 15:05:15.374445 15865 leader_election.cc:302] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c712fcc4fe7740099001b5baffdf132b, cbcfd79ba9eb489990091414aca5d91c; no voters:
I20260927 15:05:15.374689 16266 raft_consensus.cc:2781] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:05:15.374794 16266 raft_consensus.cc:482] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:05:15.374851 16266 raft_consensus.cc:3037] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:05:15.375859 16266 raft_consensus.cc:504] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.376247 16266 leader_election.cc:288] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 1 election: Requested vote from peers 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015), c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343)
I20260927 15:05:15.376565 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac52a6eaa84a38b7402520b367dd7e" candidate_uuid: "cbcfd79ba9eb489990091414aca5d91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43743b3a4f20406085872fdfa9025df8"
I20260927 15:05:15.376653 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac52a6eaa84a38b7402520b367dd7e" candidate_uuid: "cbcfd79ba9eb489990091414aca5d91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c712fcc4fe7740099001b5baffdf132b"
I20260927 15:05:15.376691 16197 raft_consensus.cc:3037] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:05:15.376772 16063 raft_consensus.cc:3037] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:05:15.378026 16063 raft_consensus.cc:2454] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbcfd79ba9eb489990091414aca5d91c in term 1.
I20260927 15:05:15.378026 16197 raft_consensus.cc:2454] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbcfd79ba9eb489990091414aca5d91c in term 1.
I20260927 15:05:15.378324 15865 leader_election.cc:302] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c712fcc4fe7740099001b5baffdf132b, cbcfd79ba9eb489990091414aca5d91c; no voters:
I20260927 15:05:15.378515 16266 raft_consensus.cc:2781] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:05:15.378736 16266 raft_consensus.cc:686] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 1 LEADER]: Becoming Leader. State: Replica: cbcfd79ba9eb489990091414aca5d91c, State: Running, Role: LEADER
I20260927 15:05:15.379043 16266 consensus_queue.cc:237] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:15.382084 15782 catalog_manager.cc:5495] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c reported cstate change: term changed from 0 to 1, leader changed from <none> to cbcfd79ba9eb489990091414aca5d91c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cbcfd79ba9eb489990091414aca5d91c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } health_report { overall_health: UNKNOWN } } }
I20260927 15:05:15.456463 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.453201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.98.1:37025)
0927 15:05:15.453329 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:15.453332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:15.453344 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:05:15.453419 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:15.453421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:15.453478 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:15.453611 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:05:15.453869 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.454356 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.455182 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:15.455395 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:15.455476 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:15.455738 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:05:15.456162 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:05:15.456262 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:15.456306 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 15:05:15.456355 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:05:15.462430 16063 raft_consensus.cc:1261] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 1 FOLLOWER]: Refusing update from remote peer cbcfd79ba9eb489990091414aca5d91c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:05:15.462430 16197 raft_consensus.cc:1261] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Refusing update from remote peer cbcfd79ba9eb489990091414aca5d91c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:05:15.463232 16271 consensus_queue.cc:1035] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:05:15.463454 16266 consensus_queue.cc:1035] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:05:15.470372 16275 mvcc.cc:205] Tried to move back new op lower bound from 7333976127326408704 to 7333976126994825216. Current Snapshot: MvccSnapshot[applied={T|T < 7333976127326408704}]
I20260927 15:05:15.470523 16276 mvcc.cc:205] Tried to move back new op lower bound from 7333976127326408704 to 7333976126994825216. Current Snapshot: MvccSnapshot[applied={T|T < 7333976127326408704}]
I20260927 15:05:15.470897 16274 mvcc.cc:205] Tried to move back new op lower bound from 7333976127326408704 to 7333976126994825216. Current Snapshot: MvccSnapshot[applied={T|T < 7333976127326408704}]
I20260927 15:05:15.478951 15784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:05:15.479128 15784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:05:15.481437 15784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Sending DeleteTablet for 3 replicas of tablet 5fac52a6eaa84a38b7402520b367dd7e
I20260927 15:05:15.482234 15909 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac52a6eaa84a38b7402520b367dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:05:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38204
I20260927 15:05:15.482378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 15846
I20260927 15:05:15.482384 16177 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac52a6eaa84a38b7402520b367dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:05:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40120
I20260927 15:05:15.482820 16043 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac52a6eaa84a38b7402520b367dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:05:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46674
I20260927 15:05:15.483274 16285 tablet_replica.cc:330] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: stopping tablet replica
I20260927 15:05:15.483600 16285 raft_consensus.cc:2229] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:05:15.483747 16284 tablet_replica.cc:330] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: stopping tablet replica
I20260927 15:05:15.483928 16285 raft_consensus.cc:2258] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:05:15.484025 16284 raft_consensus.cc:2229] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:05:15.484299 16284 raft_consensus.cc:2258] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:05:15.485456 16285 ts_tablet_manager.cc:1905] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:05:15.485730 16284 ts_tablet_manager.cc:1905] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:05:15.488229 16284 ts_tablet_manager.cc:1918] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:05:15.488229 16285 ts_tablet_manager.cc:1918] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:05:15.488332 16285 log.cc:1196] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/5fac52a6eaa84a38b7402520b367dd7e
I20260927 15:05:15.488332 16284 log.cc:1196] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/5fac52a6eaa84a38b7402520b367dd7e
I20260927 15:05:15.488656 16285 ts_tablet_manager.cc:1939] T 5fac52a6eaa84a38b7402520b367dd7e P c712fcc4fe7740099001b5baffdf132b: Deleting consensus metadata
I20260927 15:05:15.488664 16284 ts_tablet_manager.cc:1939] T 5fac52a6eaa84a38b7402520b367dd7e P 43743b3a4f20406085872fdfa9025df8: Deleting consensus metadata
W20260927 15:05:15.489373 15770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37025 (error 108)
I20260927 15:05:15.489670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 15980
I20260927 15:05:15.490249 15769 catalog_manager.cc:4841] TS 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015): tablet 5fac52a6eaa84a38b7402520b367dd7e (table test-table [id=926003d489274fc5a009c57aa34bc559]) successfully deleted
I20260927 15:05:15.490777 15771 catalog_manager.cc:4841] TS c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343): tablet 5fac52a6eaa84a38b7402520b367dd7e (table test-table [id=926003d489274fc5a009c57aa34bc559]) successfully deleted
I20260927 15:05:15.491511 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:15.491133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:38228)
0927 15:05:15.491333 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:15.491409 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 15:05:15.491712 15770 catalog_manager.cc:4584] TS cbcfd79ba9eb489990091414aca5d91c (127.0.98.1:37025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fac52a6eaa84a38b7402520b367dd7e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
I20260927 15:05:15.495483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 16114
I20260927 15:05:15.500826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 15752
I20260927 15:05:35.507490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--webserver_interface=127.0.98.62
--webserver_port=42103
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:05:35.619594 16287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:35.619915 16287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:35.619998 16287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:35.624138 16287 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:05:35.624262 16287 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:05:35.624292 16287 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:05:35.624323 16287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:05:35.624362 16287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:35.624405 16287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:35.624470 16287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:05:35.624507 16287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:05:35.629814 16287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:35.630903 16287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:35.631939 16287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:35.645941 16294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:35.646008 16293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:35.646039 16287 server_base.cc:1034] running on GCE node
W20260927 15:05:35.645974 16296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:35.646927 16287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:35.648228 16287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:35.649405 16287 hybrid_clock.cc:648] HybridClock initialized: now 1790521535649389 us; error 30 us; skew 500 ppm
Sep 27 15:05:35 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:35.652539 16287 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:05:35.653610 16287 webserver.cc:460] Webserver started at http://127.0.98.62:42103/ using document root <none> and password file <none>
I20260927 15:05:35.654282 16287 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:35.654390 16287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:35.658864 16287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:05:35.661160 16303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:35.662784 16287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:05:35.662956 16287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "278949155b6040eaa593d71530085b11"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "14ec7c63f08a579ff03469cb31330215"
server_key_iv: "507e2ee9cd8612ef00d14200d5242355"
server_key_version: "encryptionkey@0"
I20260927 15:05:35.663730 16287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:35.677068 16287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:35.680454 16287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:35.680694 16287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:35.687945 16287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33065
I20260927 15:05:35.688254 16355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33065 every 8 connection(s)
I20260927 15:05:35.689107 16287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:05:35.694025 16356 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:05:35.694406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 16287
I20260927 15:05:35.694919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37025
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33065
--webserver_port=46867
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:05:35.696017 16356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Bootstrap starting.
I20260927 15:05:35.715637 16356 log.cc:824] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:35.724895 16356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:05:35.725310 16356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Bootstrap complete.
I20260927 15:05:35.729913 16356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:35.730382 16356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278949155b6040eaa593d71530085b11, State: Initialized, Role: FOLLOWER
I20260927 15:05:35.730971 16356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:35.731135 16356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:05:35.731238 16356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:05:35.731380 16356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:05:35.733573 16356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:35.733994 16356 leader_election.cc:302] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278949155b6040eaa593d71530085b11; no voters:
I20260927 15:05:35.734360 16356 leader_election.cc:288] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:05:35.734468 16360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:05:35.734747 16360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 LEADER]: Becoming Leader. State: Replica: 278949155b6040eaa593d71530085b11, State: Running, Role: LEADER
I20260927 15:05:35.735122 16360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:35.735744 16356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:05:35.736119 16360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278949155b6040eaa593d71530085b11. Latest consensus state: current_term: 2 leader_uuid: "278949155b6040eaa593d71530085b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260927 15:05:35.737015 16361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "278949155b6040eaa593d71530085b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260927 15:05:35.737154 16360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: This master's current role is: LEADER
I20260927 15:05:35.737434 16361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: This master's current role is: LEADER
I20260927 15:05:35.741158 16376 catalog_manager.cc:1259] Loaded cluster ID: c2e0bf845ac643f6b73ad1ea1eecd565
I20260927 15:05:35.741222 16376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: loading cluster ID for follower catalog manager: success
I20260927 15:05:35.743985 16376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: acquiring CA information for follower catalog manager: success
I20260927 15:05:35.745503 16376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:05:35.745642 16369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:05:35.747784 16369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:05:35.748237 16369 catalog_manager.cc:1259] Loaded cluster ID: c2e0bf845ac643f6b73ad1ea1eecd565
I20260927 15:05:35.748317 16369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:05:35.749305 16369 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:05:35.749748 16369 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Loaded TSK: 0
I20260927 15:05:35.758775 16369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Generated new TSK 1
I20260927 15:05:35.759056 16369 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:05:35.820269 16358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:35.820557 16358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:35.820632 16358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:35.824685 16358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:35.824770 16358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:35.824863 16358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:05:35.829949 16358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46867
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:35.830986 16358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:35.832086 16358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:35.842517 16383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:35.842607 16386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:35.842517 16384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:35.843112 16358 server_base.cc:1034] running on GCE node
I20260927 15:05:35.843526 16358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:35.844132 16358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:35.845371 16358 hybrid_clock.cc:648] HybridClock initialized: now 1790521535845333 us; error 55 us; skew 500 ppm
Sep 27 15:05:35 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:35.848570 16358 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:05:35.849578 16358 webserver.cc:460] Webserver started at http://127.0.98.1:46867/ using document root <none> and password file <none>
I20260927 15:05:35.850327 16358 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:35.850440 16358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:35.855671 16358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:05:35.858430 16393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:35.859999 16358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:05:35.860128 16358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "cbcfd79ba9eb489990091414aca5d91c"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "5748cb29ace45d296111cf057af7cd30"
server_key_iv: "eafdd0df4a623d743a60301556b4222d"
server_key_version: "encryptionkey@0"
I20260927 15:05:35.860852 16358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:35.884161 16358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:35.888186 16358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:35.888373 16358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:35.889062 16358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:35.890513 16400 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:05:35.895283 16358 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:05:35.895354 16358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260927 15:05:35.895421 16358 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:05:35.897423 16358 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:05:35.897475 16358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:05:35.897997 16400 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Bootstrap starting.
I20260927 15:05:35.906970 16358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37025
I20260927 15:05:35.907302 16507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37025 every 8 connection(s)
I20260927 15:05:35.908190 16358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:05:35.912776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 16358
I20260927 15:05:35.913477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45343
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33065
--webserver_port=41249
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:05:35 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:05:35.920958 16400 log.cc:824] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:35.925947 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:35.910329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49781 (local address 127.0.98.62:33065)
0927 15:05:35.910696 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:35.910712 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:35.911123 (+ 411us) server_negotiation.cc:407] Connection header received
0927 15:05:35.912269 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:35.912280 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:35.912577 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:35.912907 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:35.914178 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.915297 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.917360 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.917961 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.918150 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:35.920712 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:35.920717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:35.920730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:35.920758 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:35.922800 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:35.923326 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:35.923337 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:35.923341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:35.923459 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:35.923794 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:35.923799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:35.923801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:35.924141 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:35.924286 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:35.924805 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 15:05:35.925059 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":146,"threads_started":1}
I20260927 15:05:35.925958 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:35.910431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:49781)
0927 15:05:35.910962 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:35.911003 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:35.911814 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:35.912971 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:35.912979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:35.913423 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:35.914009 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:35.914024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.915498 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.915502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:35.917215 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:35.917229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.918118 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.918122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:35.918324 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:35.918967 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:35.918984 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:35.920504 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:35.922927 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:35.922936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:35.922949 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:35.923204 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:35.923560 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:35.923565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:35.923566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:35.923701 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:35.924259 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:35.924265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:35.924515 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 15:05:35.924697 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 15:05:35.925027 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":159,"threads_started":1}
I20260927 15:05:35.928373 16508 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260927 15:05:35.928607 16508 heartbeater.cc:464] Registering TS with master...
I20260927 15:05:35.929096 16508 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260927 15:05:35.931444 16320 ts_manager.cc:194] Registered new tserver with Master: cbcfd79ba9eb489990091414aca5d91c (127.0.98.1:37025)
I20260927 15:05:35.931723 16400 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:05:35.932236 16400 tablet_bootstrap.cc:492] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Bootstrap complete.
I20260927 15:05:35.933940 16400 ts_tablet_manager.cc:1397] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.020s
I20260927 15:05:35.935120 16320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49781
I20260927 15:05:35.937913 16400 raft_consensus.cc:348] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:35.938745 16400 raft_consensus.cc:729] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbcfd79ba9eb489990091414aca5d91c, State: Initialized, Role: FOLLOWER
I20260927 15:05:35.939353 16400 consensus_queue.cc:260] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } }
I20260927 15:05:35.940130 16508 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260927 15:05:35.941023 16400 ts_tablet_manager.cc:1428] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
Sep 27 15:05:35 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:05:35.947147 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:35.934376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:34902)
0927 15:05:35.935295 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:35.935337 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:35.935471 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:35.936824 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:35.936830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:35.937265 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:35.937394 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:35.937403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.938625 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.938630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:35.939401 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:35.939411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.940116 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.940135 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:35.940242 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:35.940923 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:35.940937 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:35.942791 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:35.945390 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:35.945398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:35.945401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:35.945627 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:35.946020 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:35.946023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:35.946025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:35.946112 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:35.946649 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:35.946654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:35.946750 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 15:05:35.946862 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:05:35.947012 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":122,"threads_started":1}
I20260927 15:05:35.947346 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:35.935105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.0.98.1:37025)
0927 15:05:35.935748 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:35.935760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:35.935782 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:05:35.935967 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:35.935972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:35.936656 (+ 684us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:35.937560 (+ 904us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:35.937664 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.938511 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.939519 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:35.939991 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:35.940527 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:35.942971 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:35.942983 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:35.942998 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:35.943030 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:35.945264 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:35.945749 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:35.945756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:35.945758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:35.945908 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:35.946213 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:35.946217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:35.946219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:35.946529 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:35.946683 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:35.946984 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 15:05:35.947154 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":116,"threads_started":1}
I20260927 15:05:35.948527 16442 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac52a6eaa84a38b7402520b367dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:05:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34902
I20260927 15:05:35.949337 16517 tablet_replica.cc:330] stopping tablet replica
I20260927 15:05:35.949584 16517 raft_consensus.cc:2229] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:05:35.949887 16517 raft_consensus.cc:2258] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:05:35.950822 16517 ts_tablet_manager.cc:1905] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:05:35.954422 16517 ts_tablet_manager.cc:1918] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:05:35.954509 16517 log.cc:1196] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal/wals/5fac52a6eaa84a38b7402520b367dd7e
I20260927 15:05:35.954710 16517 ts_tablet_manager.cc:1939] T 5fac52a6eaa84a38b7402520b367dd7e P cbcfd79ba9eb489990091414aca5d91c: Deleting consensus metadata
I20260927 15:05:35.956831 16306 catalog_manager.cc:4841] TS cbcfd79ba9eb489990091414aca5d91c (127.0.98.1:37025): tablet 5fac52a6eaa84a38b7402520b367dd7e (table test-table [id=926003d489274fc5a009c57aa34bc559]) successfully deleted
W20260927 15:05:36.045994 16512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:36.046309 16512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:36.046403 16512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:36.050500 16512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:36.050590 16512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:36.050710 16512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:05:36.056237 16512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41249
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:36.057348 16512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:36.058395 16512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:36.068487 16523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:36.068470 16525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:36.068528 16522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:36.068909 16512 server_base.cc:1034] running on GCE node
I20260927 15:05:36.069299 16512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:36.069988 16512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:36.071153 16512 hybrid_clock.cc:648] HybridClock initialized: now 1790521536071135 us; error 31 us; skew 500 ppm
Sep 27 15:05:36 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:36.074059 16512 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:05:36.075004 16512 webserver.cc:460] Webserver started at http://127.0.98.2:41249/ using document root <none> and password file <none>
I20260927 15:05:36.075541 16512 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:36.075613 16512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:36.079368 16512 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 15:05:36.081650 16532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:36.082923 16512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:05:36.083033 16512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "c712fcc4fe7740099001b5baffdf132b"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cbcbc05e4b1b0a8f52740e719266ccf8"
server_key_iv: "56340a34ae8dbfdc938fb3ef12bedc5e"
server_key_version: "encryptionkey@0"
I20260927 15:05:36.083599 16512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:36.102191 16512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:36.105041 16512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:36.105190 16512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:36.105716 16512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:36.106531 16512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:05:36.106577 16512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:36.106616 16512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:05:36.106634 16512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:36.115507 16512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45343
I20260927 15:05:36.115526 16645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45343 every 8 connection(s)
I20260927 15:05:36.116529 16512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:05:36.122694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 16512
I20260927 15:05:36.123207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42015
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33065
--webserver_port=39087
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:05:36 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:05:36.131255 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.118340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56289 (local address 127.0.98.62:33065)
0927 15:05:36.118484 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.118488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.119516 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 15:05:36.120327 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.120329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.120374 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.120452 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:36.121594 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.122173 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.123926 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.124171 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.124257 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.126913 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:36.126916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:36.126918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:36.126944 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:36.128782 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.129419 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.129423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.129426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.129518 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.129999 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.130004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.130006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.130277 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:36.130402 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.130923 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.131068 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:05:36.132251 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.118722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:56289)
0927 15:05:36.119336 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.119374 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:36.120147 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:36.120612 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:36.120618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:36.120983 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:36.121451 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.121472 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.122311 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.122319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.123770 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.123785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.124335 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.124342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.124589 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.125284 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:36.125300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:36.126733 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:36.128983 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:36.128989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:36.129005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:36.129293 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:36.129629 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:36.129632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:36.129634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:36.129862 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:36.130391 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:36.130398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:36.130672 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 15:05:36.130879 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 15:05:36.131318 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":105,"threads_started":1}
I20260927 15:05:36.133589 16646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260927 15:05:36.133894 16646 heartbeater.cc:464] Registering TS with master...
I20260927 15:05:36.134431 16646 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260927 15:05:36.135540 16320 ts_manager.cc:194] Registered new tserver with Master: c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343)
I20260927 15:05:36.136323 16320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56289
W20260927 15:05:36.235929 16649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:36.236271 16649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:36.236321 16649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:36.240679 16649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:36.240736 16649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:36.240830 16649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:05:36.246098 16649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39087
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:36.247118 16649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:36.248092 16649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:36.257448 16655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:36.257591 16654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:36.257501 16657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:36.257838 16649 server_base.cc:1034] running on GCE node
I20260927 15:05:36.258287 16649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:36.258912 16649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:36.260077 16649 hybrid_clock.cc:648] HybridClock initialized: now 1790521536260046 us; error 51 us; skew 500 ppm
Sep 27 15:05:36 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:36.262799 16649 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:05:36.263602 16649 webserver.cc:460] Webserver started at http://127.0.98.3:39087/ using document root <none> and password file <none>
I20260927 15:05:36.264096 16649 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:36.264163 16649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:36.268260 16649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 15:05:36.270665 16664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:36.272019 16649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:05:36.272115 16649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "43743b3a4f20406085872fdfa9025df8"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cc1a7276a1f459e810e5a94084aae72e"
server_key_iv: "80a2a88625d2d2c2d9f8967cf09b68b5"
server_key_version: "encryptionkey@0"
I20260927 15:05:36.272712 16649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:36.284518 16649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:36.287775 16649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:36.287963 16649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:36.288504 16649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:36.289343 16649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:05:36.289387 16649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:36.289426 16649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:05:36.289443 16649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:36.298632 16649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42015
I20260927 15:05:36.298668 16777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42015 every 8 connection(s)
I20260927 15:05:36.299602 16649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:05:36.300091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 16649
I20260927 15:05:36.310135 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.300857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.0.98.62:33065)
0927 15:05:36.300983 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.300987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.301294 (+ 307us) server_negotiation.cc:407] Connection header received
0927 15:05:36.301506 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.301512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.301581 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.301671 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:36.302589 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.303284 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.304487 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.304723 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.304808 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.306330 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:36.306334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:36.306336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:36.306372 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:36.308145 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.308719 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.308722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.308723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.308802 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.309120 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.309124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.309127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.309336 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:36.309547 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.309742 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.309913 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 15:05:36 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:05:36.314860 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.301783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51097 (local address 127.0.98.62:33065)
0927 15:05:36.302022 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.302027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.303071 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 15:05:36.304171 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.304174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.304220 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.304318 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:36.305829 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.306401 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.307813 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.308236 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.308337 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.311242 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:36.311245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:36.311248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:36.311270 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:36.312937 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.313467 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.313469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.313471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.313546 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.313914 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.313917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.313918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.314115 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:36.314194 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.314649 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.314743 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260927 15:05:36.315718 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.302261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:51097)
0927 15:05:36.302940 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.302978 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:36.303940 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:36.304532 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:36.304538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:36.304977 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:36.305624 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.305646 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.306545 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.306547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.307688 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.307697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.308393 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.308408 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.308690 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.309252 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:36.309267 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:36.311054 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:36.313058 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:36.313063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:36.313074 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:36.313325 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:36.313631 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:36.313633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:36.313634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:36.313824 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:36.314203 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:36.314208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:36.314430 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 15:05:36.314575 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 15:05:36.314810 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":115,"threads_started":1}
I20260927 15:05:36.316794 16778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260927 15:05:36.317075 16778 heartbeater.cc:464] Registering TS with master...
I20260927 15:05:36.317545 16778 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260927 15:05:36.318603 16320 ts_manager.cc:194] Registered new tserver with Master: 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015)
I20260927 15:05:36.319168 16320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51097
I20260927 15:05:36.324040 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:05:36.328500 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.324687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.0.98.62:33065)
0927 15:05:36.324805 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.324809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.325002 (+ 193us) server_negotiation.cc:407] Connection header received
0927 15:05:36.325168 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.325170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.325259 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.325350 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:05:36.325678 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.326289 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.327289 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.327515 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.327596 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.327952 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:05:36.328097 (+ 145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 15:05:36.328340 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48108: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 15:05:36.328563 16781 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48108: token expired
W20260927 15:05:36.328866 16247 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33065)
I20260927 15:05:36.336954 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.329255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.0.98.62:33065)
0927 15:05:36.329382 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.329386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.329450 (+ 64us) server_negotiation.cc:407] Connection header received
0927 15:05:36.329490 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.329492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.329529 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.329601 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:36.330340 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.330856 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.331683 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.331886 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.331971 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.333522 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:36.333525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:36.333527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:36.333551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:36.335162 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.335755 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.335758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.335760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.335838 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:36.336089 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:36.336091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:36.336092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:36.336251 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:36.336403 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.336684 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.336786 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:05:36.339679 16320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:05:36.342924 16320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:05:36.351061 16442 tablet_service.cc:1467] Processing CreateTablet for tablet a4e4d4fea5644217bedac9544eeb63c4 (DEFAULT_TABLE table=test-table [id=71563fe6406d4f49843cb62be3ffad4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:05:36.351742 16442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e4d4fea5644217bedac9544eeb63c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:05:36.354429 16400 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Bootstrap starting.
I20260927 15:05:36.355396 16400 tablet_bootstrap.cc:654] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Neither blocks nor log segments found. Creating new log.
I20260927 15:05:36.355858 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.349713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45343 (local address 127.0.0.1:54444)
0927 15:05:36.349919 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.349936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:36.350028 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:36.350880 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:36.350884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:36.350907 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:36.351027 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.351036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.352968 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.352976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.354127 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.354138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.355421 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.355425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.355529 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.355582 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 15:05:36.355640 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:05:36.355720 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":35}
I20260927 15:05:36.356576 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.349918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42015 (local address 127.0.0.1:33300)
0927 15:05:36.350494 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.350508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:36.350591 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:36.351474 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:36.351477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:36.351497 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:36.351589 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.351595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.353047 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.353051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.354883 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.354895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.356166 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.356171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.356269 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.356287 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:05:36.356378 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:05:36.356458 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":102,"threads_started":1}
I20260927 15:05:36.356648 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.350497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.98.3:42015)
0927 15:05:36.350971 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.350983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.351007 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:05:36.351089 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.351095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.351318 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.351496 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:36.351678 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.352920 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.355025 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.355843 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.355951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.356093 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.356416 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.356509 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":116,"threads_started":1}
I20260927 15:05:36.357007 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.349940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.0.98.2:45343)
0927 15:05:36.350280 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.350294 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.350320 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:05:36.350421 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.350428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.350682 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.351217 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:36.351442 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.352807 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.354268 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.355160 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.356339 (+ 1179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.356531 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.356593 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.356846 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":139,"threads_started":1}
I20260927 15:05:36.357056 16400 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: No bootstrap required, opened a new log
I20260927 15:05:36.357134 16400 ts_tablet_manager.cc:1397] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 15:05:36.357609 16400 raft_consensus.cc:348] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.357733 16400 raft_consensus.cc:374] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:05:36.357811 16400 raft_consensus.cc:729] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbcfd79ba9eb489990091414aca5d91c, State: Initialized, Role: FOLLOWER
I20260927 15:05:36.357966 16400 consensus_queue.cc:260] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.358263 16400 ts_tablet_manager.cc:1428] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:05:36.359947 16712 tablet_service.cc:1467] Processing CreateTablet for tablet a4e4d4fea5644217bedac9544eeb63c4 (DEFAULT_TABLE table=test-table [id=71563fe6406d4f49843cb62be3ffad4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:05:36.360052 16580 tablet_service.cc:1467] Processing CreateTablet for tablet a4e4d4fea5644217bedac9544eeb63c4 (DEFAULT_TABLE table=test-table [id=71563fe6406d4f49843cb62be3ffad4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:05:36.361092 16580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e4d4fea5644217bedac9544eeb63c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:05:36.361105 16712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e4d4fea5644217bedac9544eeb63c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:05:36.368147 16791 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Bootstrap starting.
I20260927 15:05:36.368319 16792 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Bootstrap starting.
I20260927 15:05:36.370417 16791 tablet_bootstrap.cc:654] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Neither blocks nor log segments found. Creating new log.
I20260927 15:05:36.370919 16792 tablet_bootstrap.cc:654] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Neither blocks nor log segments found. Creating new log.
I20260927 15:05:36.371676 16791 log.cc:824] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:36.371747 16792 log.cc:824] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:36.373493 16792 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: No bootstrap required, opened a new log
I20260927 15:05:36.373505 16791 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: No bootstrap required, opened a new log
I20260927 15:05:36.373698 16791 ts_tablet_manager.cc:1397] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:05:36.373817 16792 ts_tablet_manager.cc:1397] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:05:36.376873 16791 raft_consensus.cc:348] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.376889 16792 raft_consensus.cc:348] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.377156 16791 raft_consensus.cc:374] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:05:36.377179 16792 raft_consensus.cc:374] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:05:36.377223 16791 raft_consensus.cc:729] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43743b3a4f20406085872fdfa9025df8, State: Initialized, Role: FOLLOWER
I20260927 15:05:36.377252 16792 raft_consensus.cc:729] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c712fcc4fe7740099001b5baffdf132b, State: Initialized, Role: FOLLOWER
I20260927 15:05:36.377642 16791 consensus_queue.cc:260] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.377830 16792 consensus_queue.cc:260] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.378374 16778 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260927 15:05:36.378640 16791 ts_tablet_manager.cc:1428] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:05:36.378818 16792 ts_tablet_manager.cc:1428] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:05:36.379124 16646 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260927 15:05:36.385046 16796 raft_consensus.cc:482] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:05:36.385231 16796 raft_consensus.cc:504] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.386381 16796 leader_election.cc:288] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015), cbcfd79ba9eb489990091414aca5d91c (127.0.98.1:37025)
I20260927 15:05:36.390465 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.386778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54007 (local address 127.0.98.3:42015)
0927 15:05:36.386935 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.386938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.386952 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:05:36.387059 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.387062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.387112 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.387199 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:36.387516 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.388090 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.389378 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.389952 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.390034 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.390084 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.390311 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.390367 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:05:36.390493 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.386566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42015 (local address 127.0.98.2:54007)
0927 15:05:36.386818 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.386849 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:36.386940 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:36.387226 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:36.387229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:36.387244 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:36.387356 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.387384 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.388193 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.388197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.389231 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.389241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.390066 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.390069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.390179 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.390205 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:05:36.390292 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:05:36.390366 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260927 15:05:36.391233 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.386774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50085 (local address 127.0.98.1:37025)
0927 15:05:36.387092 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.387096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.387111 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:05:36.387225 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.387228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.387291 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.387384 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:05:36.387687 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.388500 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.389671 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.390458 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.390575 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.390746 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.391075 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.391144 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260927 15:05:36.391245 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.386668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.98.2:50085)
0927 15:05:36.387020 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.387036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:36.387120 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:36.387416 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:36.387422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:36.387444 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:05:36.387569 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.387574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.388623 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.388626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.389527 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:36.389539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.390820 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.390824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:36.390933 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.390965 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:05:36.391044 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:05:36.391115 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20260927 15:05:36.391214 16732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e4d4fea5644217bedac9544eeb63c4" candidate_uuid: "c712fcc4fe7740099001b5baffdf132b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43743b3a4f20406085872fdfa9025df8" is_pre_election: true
I20260927 15:05:36.391494 16732 raft_consensus.cc:2454] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c712fcc4fe7740099001b5baffdf132b in term 0.
I20260927 15:05:36.392006 16534 leader_election.cc:302] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43743b3a4f20406085872fdfa9025df8, c712fcc4fe7740099001b5baffdf132b; no voters:
I20260927 15:05:36.392309 16796 raft_consensus.cc:2781] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:05:36.392258 16462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e4d4fea5644217bedac9544eeb63c4" candidate_uuid: "c712fcc4fe7740099001b5baffdf132b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbcfd79ba9eb489990091414aca5d91c" is_pre_election: true
I20260927 15:05:36.392393 16796 raft_consensus.cc:482] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:05:36.392480 16796 raft_consensus.cc:3037] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:05:36.392565 16462 raft_consensus.cc:2454] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c712fcc4fe7740099001b5baffdf132b in term 0.
I20260927 15:05:36.393643 16796 raft_consensus.cc:504] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.394011 16796 leader_election.cc:288] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [CANDIDATE]: Term 1 election: Requested vote from peers 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015), cbcfd79ba9eb489990091414aca5d91c (127.0.98.1:37025)
I20260927 15:05:36.394351 16462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e4d4fea5644217bedac9544eeb63c4" candidate_uuid: "c712fcc4fe7740099001b5baffdf132b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbcfd79ba9eb489990091414aca5d91c"
I20260927 15:05:36.394459 16462 raft_consensus.cc:3037] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:05:36.395511 16462 raft_consensus.cc:2454] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c712fcc4fe7740099001b5baffdf132b in term 1.
I20260927 15:05:36.395761 16732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e4d4fea5644217bedac9544eeb63c4" candidate_uuid: "c712fcc4fe7740099001b5baffdf132b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43743b3a4f20406085872fdfa9025df8"
I20260927 15:05:36.395833 16535 leader_election.cc:302] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c712fcc4fe7740099001b5baffdf132b, cbcfd79ba9eb489990091414aca5d91c; no voters:
I20260927 15:05:36.395886 16732 raft_consensus.cc:3037] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:05:36.396004 16796 raft_consensus.cc:2781] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:05:36.396260 16796 raft_consensus.cc:686] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 1 LEADER]: Becoming Leader. State: Replica: c712fcc4fe7740099001b5baffdf132b, State: Running, Role: LEADER
I20260927 15:05:36.396611 16796 consensus_queue.cc:237] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:36.397192 16732 raft_consensus.cc:2454] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c712fcc4fe7740099001b5baffdf132b in term 1.
I20260927 15:05:36.399792 16320 catalog_manager.cc:5495] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b reported cstate change: term changed from 0 to 1, leader changed from <none> to c712fcc4fe7740099001b5baffdf132b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c712fcc4fe7740099001b5baffdf132b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: UNKNOWN } } }
W20260927 15:05:36.409498 16509 tablet.cc:2369] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:05:36.411563 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.407936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.0.98.2:45343)
0927 15:05:36.408073 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:36.408077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:36.408094 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:05:36.408176 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:36.408179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:36.408233 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:36.408330 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:05:36.408578 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.409095 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.410060 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:36.410304 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:36.410401 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:36.410719 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:05:36.411167 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:05:36.411276 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:36.411368 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:05:36.411424 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:05:36.417431 16462 raft_consensus.cc:1261] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Refusing update from remote peer c712fcc4fe7740099001b5baffdf132b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:05:36.417438 16732 raft_consensus.cc:1261] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Refusing update from remote peer c712fcc4fe7740099001b5baffdf132b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:05:36.418133 16798 consensus_queue.cc:1035] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:05:36.418326 16796 consensus_queue.cc:1035] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:05:36.423686 16801 mvcc.cc:205] Tried to move back new op lower bound from 7333976213159112704 to 7333976213083549696. Current Snapshot: MvccSnapshot[applied={T|T < 7333976213159112704}]
I20260927 15:05:36.424674 16802 mvcc.cc:205] Tried to move back new op lower bound from 7333976213159112704 to 7333976213083549696. Current Snapshot: MvccSnapshot[applied={T|T < 7333976213159112704}]
I20260927 15:05:36.428964 16804 mvcc.cc:205] Tried to move back new op lower bound from 7333976213159112704 to 7333976213083549696. Current Snapshot: MvccSnapshot[applied={T|T < 7333976213159112704}]
I20260927 15:05:36.432449 16320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:05:36.432637 16320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:05:36.435189 16320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Sending DeleteTablet for 3 replicas of tablet a4e4d4fea5644217bedac9544eeb63c4
I20260927 15:05:36.436090 16712 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e4d4fea5644217bedac9544eeb63c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:05:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33300
I20260927 15:05:36.436064 16580 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e4d4fea5644217bedac9544eeb63c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:05:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54444
I20260927 15:05:36.436251 16442 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e4d4fea5644217bedac9544eeb63c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:05:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34902
I20260927 15:05:36.436435 16517 tablet_replica.cc:330] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: stopping tablet replica
I20260927 15:05:36.436542 16517 raft_consensus.cc:2229] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:05:36.436638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 16358
I20260927 15:05:36.436693 16517 raft_consensus.cc:2258] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:05:36.438779 16812 tablet_replica.cc:330] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: stopping tablet replica
I20260927 15:05:36.439204 16812 raft_consensus.cc:2229] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:05:36.439594 16812 raft_consensus.cc:2258] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:05:36.440030 16811 tablet_replica.cc:330] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: stopping tablet replica
I20260927 15:05:36.440369 16811 raft_consensus.cc:2229] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:05:36.440779 16811 raft_consensus.cc:2258] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:05:36.441676 16812 ts_tablet_manager.cc:1905] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:05:36.442116 16811 ts_tablet_manager.cc:1905] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:05:36.444638 16812 ts_tablet_manager.cc:1918] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:05:36.444737 16812 log.cc:1196] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/a4e4d4fea5644217bedac9544eeb63c4
W20260927 15:05:36.444865 16306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37025 (error 108)
I20260927 15:05:36.445078 16812 ts_tablet_manager.cc:1939] T a4e4d4fea5644217bedac9544eeb63c4 P 43743b3a4f20406085872fdfa9025df8: Deleting consensus metadata
I20260927 15:05:36.445544 16811 ts_tablet_manager.cc:1918] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:05:36.445698 16811 log.cc:1196] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/a4e4d4fea5644217bedac9544eeb63c4
I20260927 15:05:36.446074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 16512
I20260927 15:05:36.446116 16811 ts_tablet_manager.cc:1939] T a4e4d4fea5644217bedac9544eeb63c4 P c712fcc4fe7740099001b5baffdf132b: Deleting consensus metadata
I20260927 15:05:36.446694 16305 catalog_manager.cc:4841] TS 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015): tablet a4e4d4fea5644217bedac9544eeb63c4 (table test-table [id=71563fe6406d4f49843cb62be3ffad4d]) successfully deleted
I20260927 15:05:36.448429 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.448069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:34906)
0927 15:05:36.448240 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.448321 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:05:36.448671 16306 catalog_manager.cc:4584] TS cbcfd79ba9eb489990091414aca5d91c (127.0.98.1:37025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4e4d4fea5644217bedac9544eeb63c4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
I20260927 15:05:36.452879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 16649
I20260927 15:05:36.456656 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:36.453153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45343 (local address 127.0.0.1:54458)
0927 15:05:36.456527 (+ 3374us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:36.456583 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3286}
I20260927 15:05:36.462359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 16287
I20260927 15:05:56.469249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--webserver_interface=127.0.98.62
--webserver_port=42103
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:05:56.580453 16814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:56.580766 16814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:56.580837 16814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:56.584854 16814 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:05:56.584949 16814 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:05:56.584975 16814 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:05:56.584996 16814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:05:56.585035 16814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:56.585076 16814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:56.585129 16814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:05:56.585170 16814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:05:56.590396 16814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:56.591456 16814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:56.592453 16814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:56.601115 16820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:56.601104 16822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:56.601104 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:56.601302 16814 server_base.cc:1034] running on GCE node
I20260927 15:05:56.602240 16814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:56.603348 16814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:56.604516 16814 hybrid_clock.cc:648] HybridClock initialized: now 1790521556604504 us; error 25 us; skew 500 ppm
Sep 27 15:05:56 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:56.607340 16814 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:05:56.608374 16814 webserver.cc:460] Webserver started at http://127.0.98.62:42103/ using document root <none> and password file <none>
I20260927 15:05:56.608947 16814 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:56.609009 16814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:56.612954 16814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:05:56.615252 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:56.616555 16814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:05:56.616700 16814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "278949155b6040eaa593d71530085b11"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "14ec7c63f08a579ff03469cb31330215"
server_key_iv: "507e2ee9cd8612ef00d14200d5242355"
server_key_version: "encryptionkey@0"
I20260927 15:05:56.617238 16814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:56.639734 16814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:56.642783 16814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:56.643000 16814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:56.650216 16814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33065
I20260927 15:05:56.650502 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33065 every 8 connection(s)
I20260927 15:05:56.651340 16814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:05:56.655553 16882 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:05:56.656456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 16814
I20260927 15:05:56.656929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37025
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33065
--webserver_port=46867
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:05:56.657258 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Bootstrap starting.
I20260927 15:05:56.674700 16882 log.cc:824] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:56.686661 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:05:56.687075 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Bootstrap complete.
I20260927 15:05:56.691169 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:56.691555 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278949155b6040eaa593d71530085b11, State: Initialized, Role: FOLLOWER
I20260927 15:05:56.692039 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:56.692194 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:05:56.692267 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:05:56.692380 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:05:56.694407 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:56.694741 16882 leader_election.cc:302] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278949155b6040eaa593d71530085b11; no voters:
I20260927 15:05:56.695060 16882 leader_election.cc:288] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:05:56.695142 16886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:05:56.695333 16886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [term 3 LEADER]: Becoming Leader. State: Replica: 278949155b6040eaa593d71530085b11, State: Running, Role: LEADER
I20260927 15:05:56.695617 16886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260927 15:05:56.696256 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:05:56.696544 16886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278949155b6040eaa593d71530085b11. Latest consensus state: current_term: 3 leader_uuid: "278949155b6040eaa593d71530085b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260927 15:05:56.696664 16886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: This master's current role is: LEADER
I20260927 15:05:56.697019 16890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:05:56.697044 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "278949155b6040eaa593d71530085b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278949155b6040eaa593d71530085b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260927 15:05:56.697257 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11 [sys.catalog]: This master's current role is: LEADER
I20260927 15:05:56.702332 16890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:05:56.702999 16890 catalog_manager.cc:1259] Loaded cluster ID: c2e0bf845ac643f6b73ad1ea1eecd565
I20260927 15:05:56.703061 16890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:05:56.705538 16890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:05:56.706313 16890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Loaded TSK: 0
I20260927 15:05:56.706396 16890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 278949155b6040eaa593d71530085b11: Loaded TSK: 1
F20260927 15:05:56.707113 16890 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0e021c69ad google::LogMessage::Fail() at ??:0
@ 0x7f0e021c988a google::LogMessage::SendToLog() at ??:0
@ 0x7f0e021c64de google::LogMessage::Flush() at ??:0
@ 0x7f0e021c7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0e04e8a18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0e04e82f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0e0cb2b7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0e0cb0599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0e0cb06178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f0e0cb47df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f0e0cb6682b std::function<>::operator()() at ??:0
@ 0x7f0e0cb05b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0e0cb06f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0e0cb03579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0e0cb47640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0e044183d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0e04418c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0e0441a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0e04409512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0e05a4d6db start_thread at ??:0
@ 0x7f0e00cc461f clone at ??:0
W20260927 15:05:56.789359 16884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:56.789592 16884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:56.789638 16884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:56.794176 16884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:56.794231 16884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:56.794308 16884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:05:56.800024 16884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46867
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:56.801091 16884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:56.802361 16884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:56.812664 16928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:56.812664 16927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:56.813171 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:56.813625 16884 server_base.cc:1034] running on GCE node
I20260927 15:05:56.814147 16884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:56.814910 16884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:56.816090 16884 hybrid_clock.cc:648] HybridClock initialized: now 1790521556816056 us; error 50 us; skew 500 ppm
Sep 27 15:05:56 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:56.819217 16884 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:05:56.820255 16884 webserver.cc:460] Webserver started at http://127.0.98.1:46867/ using document root <none> and password file <none>
I20260927 15:05:56.820842 16884 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:56.820900 16884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:56.825394 16884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:05:56.827904 16939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:56.829308 16884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:05:56.829490 16884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "cbcfd79ba9eb489990091414aca5d91c"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "5748cb29ace45d296111cf057af7cd30"
server_key_iv: "eafdd0df4a623d743a60301556b4222d"
server_key_version: "encryptionkey@0"
I20260927 15:05:56.830186 16884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:56.852797 16884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:56.856673 16884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:56.856886 16884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:56.857554 16884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:56.858713 16946 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:05:56.862844 16884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:05:56.862911 16884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 15:05:56.862964 16884 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:05:56.864714 16884 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:05:56.864773 16884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:05:56.865036 16946 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Bootstrap starting.
I20260927 15:05:56.875849 16884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37025
I20260927 15:05:56.876262 17053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37025 every 8 connection(s)
I20260927 15:05:56.877275 16884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:05:56.884107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 16884
I20260927 15:05:56.884722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45343
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33065
--webserver_port=41249
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:05:56.886884 16946 log.cc:824] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:05:56.896529 16946 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:05:56.897001 16946 tablet_bootstrap.cc:492] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Bootstrap complete.
I20260927 15:05:56.898633 16946 ts_tablet_manager.cc:1397] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.018s
I20260927 15:05:56.902205 16946 raft_consensus.cc:348] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:56.902797 16946 raft_consensus.cc:729] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbcfd79ba9eb489990091414aca5d91c, State: Initialized, Role: FOLLOWER
I20260927 15:05:56.903357 16946 consensus_queue.cc:260] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:56.904541 16946 ts_tablet_manager.cc:1428] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260927 15:05:56.961093 17056 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:05:56.879870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:45493)
0927 15:05:56.880483 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:56.880551 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:56.881827 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:56.960019 (+ 78192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":135,"threads_started":1}
W20260927 15:05:56.961648 17054 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33065 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33065: Client connection negotiation failed: client connection to 127.0.98.62:33065: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:05:56.962476 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:56.962154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:52047)
0927 15:05:56.962294 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:56.962394 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:05:56.963182 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:56.962994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:36403)
0927 15:05:56.963095 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:56.963136 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:05:56.963421 17054 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:05:56.996649 17057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:56.996940 17057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:56.997012 17057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:57.001000 17057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:57.001084 17057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:57.001171 17057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:05:57.006361 17057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41249
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:57.007453 17057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:57.008564 17057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:57.018544 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:57.018582 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:57.018591 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:57.018874 17057 server_base.cc:1034] running on GCE node
I20260927 15:05:57.019428 17057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:57.020107 17057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:57.021318 17057 hybrid_clock.cc:648] HybridClock initialized: now 1790521557021291 us; error 41 us; skew 500 ppm
Sep 27 15:05:57 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:57.024120 17057 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:05:57.025038 17057 webserver.cc:460] Webserver started at http://127.0.98.2:41249/ using document root <none> and password file <none>
I20260927 15:05:57.025609 17057 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:57.025699 17057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:57.029558 17057 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:05:57.031797 17074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:57.033068 17057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:05:57.033229 17057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "c712fcc4fe7740099001b5baffdf132b"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cbcbc05e4b1b0a8f52740e719266ccf8"
server_key_iv: "56340a34ae8dbfdc938fb3ef12bedc5e"
server_key_version: "encryptionkey@0"
I20260927 15:05:57.033896 17057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:57.050894 17057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:57.054028 17057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:57.054232 17057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:57.054832 17057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:57.055668 17057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:05:57.055734 17057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:57.055800 17057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:05:57.055851 17057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:57.064780 17057 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45343
I20260927 15:05:57.064805 17187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45343 every 8 connection(s)
I20260927 15:05:57.065747 17057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:05:57.069384 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:57.067968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:52501)
0927 15:05:57.068610 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:57.068714 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":108,"threads_started":1}
W20260927 15:05:57.069911 17188 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33065 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33065: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
I20260927 15:05:57.070430 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:57.070240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:45495)
0927 15:05:57.070338 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:57.070380 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 15:05:57.071019 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:57.070853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:56817)
0927 15:05:57.070936 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:57.070977 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 15:05:57.071213 17188 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:05:57.071753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 17057
I20260927 15:05:57.072227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42015
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33065
--webserver_port=39087
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:05:57.184283 17191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:05:57.184561 17191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:05:57.184603 17191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:05:57.188683 17191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:05:57.188746 17191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:05:57.188832 17191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:05:57.194273 17191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39087
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:05:57.195422 17191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:05:57.196503 17191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:05:57.206208 17199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:05:57.206244 17197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:57.206202 17191 server_base.cc:1034] running on GCE node
W20260927 15:05:57.206204 17196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:05:57.206749 17191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:05:57.207435 17191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:05:57.208591 17191 hybrid_clock.cc:648] HybridClock initialized: now 1790521557208557 us; error 47 us; skew 500 ppm
Sep 27 15:05:57 dist-test-slave-ml7t krb5kdc[15739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:05:57.211369 17191 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:05:57.212272 17191 webserver.cc:460] Webserver started at http://127.0.98.3:39087/ using document root <none> and password file <none>
I20260927 15:05:57.212828 17191 fs_manager.cc:359] Metadata directory not provided
I20260927 15:05:57.212891 17191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:05:57.216817 17191 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:05:57.218854 17206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:57.220067 17191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:05:57.220187 17191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "43743b3a4f20406085872fdfa9025df8"
format_stamp: "Formatted at 2026-09-27 15:05:14 on dist-test-slave-ml7t"
server_key: "cc1a7276a1f459e810e5a94084aae72e"
server_key_iv: "80a2a88625d2d2c2d9f8967cf09b68b5"
server_key_version: "encryptionkey@0"
I20260927 15:05:57.220685 17191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:05:57.245234 17191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:05:57.248901 17191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:05:57.249126 17191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:05:57.249822 17191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:05:57.250728 17191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:05:57.250782 17191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:57.250857 17191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:05:57.250901 17191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:05:57.260681 17191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42015
I20260927 15:05:57.260704 17319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42015 every 8 connection(s)
I20260927 15:05:57.261762 17191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:05:57.265400 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:57.263945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:35617)
0927 15:05:57.264603 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:57.264708 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":116,"threads_started":1}
W20260927 15:05:57.265874 17320 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33065 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33065: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
I20260927 15:05:57.266528 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:57.266258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:45587)
0927 15:05:57.266355 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:57.266399 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 15:05:57.267104 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:57.266931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:34547)
0927 15:05:57.267020 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:57.267059 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 15:05:57.267277 17320 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:05:57.268724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 17191
I20260927 15:05:57.964532 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:57.964029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:36743)
0927 15:05:57.964318 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:57.964405 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260927 15:05:58.072163 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:58.071766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:49825)
0927 15:05:58.072003 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:58.072069 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260927 15:05:58.268179 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:58.267787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:49321)
0927 15:05:58.268022 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:58.268082 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260927 15:05:58.707458 17327 raft_consensus.cc:482] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:05:58.707674 17327 raft_consensus.cc:504] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:05:58.708770 17327 leader_election.cc:288] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343), 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015)
Sep 27 15:05:58 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 15:05:58 dist-test-slave-ml7t krb5kdc[15739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 15:05:58.727391 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:58.709029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42015 (local address 127.0.98.1:39165)
0927 15:05:58.709412 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:58.709430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:58.709549 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:58.711239 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:58.711245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:58.711845 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:58.712327 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:58.712344 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.713634 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.713641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:58.714750 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:58.714763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.715715 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.715718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:58.715869 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:58.720116 (+ 4247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:58.720128 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:58.722021 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:58.724282 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:58.724287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:58.724289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:58.724818 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:58.725216 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:58.725219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:58.725221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:58.725407 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:58.726230 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:58.726237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:58.726628 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 15:05:58.726842 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 15:05:58.727238 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":300,"spinlock_wait_cycles":9856,"thread_start_us":80,"threads_started":1}
I20260927 15:05:58.727391 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:58.709029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45343 (local address 127.0.98.1:59421)
0927 15:05:58.709587 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:58.709602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:05:58.709714 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:05:58.711239 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:05:58.711245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:05:58.711842 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:05:58.712327 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:58.712343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.713683 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.713686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:58.714736 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:05:58.714748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.715969 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.715972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:05:58.716039 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:58.720092 (+ 4053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:05:58.720119 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:05:58.721677 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:05:58.724185 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:58.724196 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:58.724207 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:58.724783 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:58.725207 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:05:58.725211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:05:58.725213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:05:58.725413 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:05:58.726230 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:05:58.726237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:05:58.726628 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 15:05:58.726842 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 15:05:58.727186 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":327,"spinlock_wait_cycles":39680,"thread_start_us":111,"threads_started":1}
I20260927 15:05:58.727468 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:58.709165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59421 (local address 127.0.98.2:45343)
0927 15:05:58.709701 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:58.709712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:58.709748 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:05:58.710507 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:58.710513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:58.710864 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:58.711167 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:58.712519 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.713533 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.714881 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.715591 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.715767 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:58.721925 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:58.721930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:58.721941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:58.721966 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:58.724052 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:58.724912 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:58.724918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:58.724923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:58.725077 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:58.725573 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:58.725579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:58.725583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:58.726093 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:58.726248 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:58.726984 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 15:05:58.727268 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":69,"threads_started":1}
I20260927 15:05:58.728039 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:58.709305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39165 (local address 127.0.98.3:42015)
0927 15:05:58.709651 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 15:05:58.709674 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:05:58.709716 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:05:58.710537 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:05:58.710544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:05:58.710865 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:05:58.711190 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:05:58.712500 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.713491 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.714911 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:05:58.715591 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:05:58.715767 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:05:58.722207 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:05:58.722215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:05:58.722230 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:05:58.722266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:05:58.724168 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:58.724912 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:58.724918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:58.724923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:58.725077 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:05:58.725529 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:05:58.725535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:05:58.725540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:05:58.726094 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:05:58.726248 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:05:58.726984 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 15:05:58.727268 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
W20260927 15:05:58.730270 16941 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
W20260927 15:05:58.730855 16943 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
I20260927 15:05:58.730950 16943 leader_election.cc:302] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbcfd79ba9eb489990091414aca5d91c; no voters: 43743b3a4f20406085872fdfa9025df8, c712fcc4fe7740099001b5baffdf132b
I20260927 15:05:58.731206 17327 raft_consensus.cc:2726] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:05:58.965561 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:58.965243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:41115)
0927 15:05:58.965422 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:58.965477 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 15:05:59.073287 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:59.072872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:36367)
0927 15:05:59.073135 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:59.073189 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260927 15:05:59.269279 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:59.268905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:59813)
0927 15:05:59.269144 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:59.269197 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260927 15:05:59.966954 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:05:59.966463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:52037)
0927 15:05:59.966742 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 15:05:59.966808 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260927 15:06:00.074501 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:00.074045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:58107)
0927 15:06:00.074303 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:00.074406 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260927 15:06:00.243620 17336 raft_consensus.cc:482] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:00.243744 17336 raft_consensus.cc:504] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:06:00.244259 17336 leader_election.cc:288] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343), 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015)
W20260927 15:06:00.245060 16941 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
W20260927 15:06:00.245169 16943 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
I20260927 15:06:00.245234 16943 leader_election.cc:302] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbcfd79ba9eb489990091414aca5d91c; no voters: 43743b3a4f20406085872fdfa9025df8, c712fcc4fe7740099001b5baffdf132b
I20260927 15:06:00.245362 17336 raft_consensus.cc:2726] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:06:00.270408 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:00.269979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:49567)
0927 15:06:00.270214 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:00.270272 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260927 15:06:00.968489 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:00.968024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:59705)
0927 15:06:00.968291 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:00.968346 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
I20260927 15:06:01.075734 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:01.075301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:54461)
0927 15:06:01.075566 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:01.075627 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260927 15:06:01.271595 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:01.271168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:44207)
0927 15:06:01.271427 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:01.271497 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260927 15:06:01.969970 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:01.969455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:57659)
0927 15:06:01.969760 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:01.969870 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260927 15:06:02.077015 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:02.076579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:48579)
0927 15:06:02.076846 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:02.076906 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260927 15:06:02.272852 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:02.272419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:32851)
0927 15:06:02.272690 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:02.272754 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260927 15:06:02.971196 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:02.970765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:42961)
0927 15:06:02.971034 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:02.971098 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260927 15:06:03.078243 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:03.077691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:34029)
0927 15:06:03.078094 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:03.078151 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":118,"threads_started":1}
I20260927 15:06:03.156495 17346 raft_consensus.cc:482] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:03.156656 17346 raft_consensus.cc:504] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:06:03.157145 17346 leader_election.cc:288] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343), 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015)
W20260927 15:06:03.158111 16941 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
W20260927 15:06:03.158238 16943 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
I20260927 15:06:03.158316 16943 leader_election.cc:302] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbcfd79ba9eb489990091414aca5d91c; no voters: 43743b3a4f20406085872fdfa9025df8, c712fcc4fe7740099001b5baffdf132b
I20260927 15:06:03.158633 17346 raft_consensus.cc:2726] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:06:03.274072 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:03.273615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:42927)
0927 15:06:03.273909 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:03.273979 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260927 15:06:03.972607 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:03.972115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:52387)
0927 15:06:03.972440 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:03.972507 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
I20260927 15:06:04.079519 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:04.079041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:55539)
0927 15:06:04.079360 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:04.079425 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":72,"threads_started":1}
I20260927 15:06:04.275290 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:04.274813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:47897)
0927 15:06:04.275136 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:04.275197 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":74,"threads_started":1}
I20260927 15:06:04.974041 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:04.973461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:34925)
0927 15:06:04.973845 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:04.973938 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260927 15:06:05.080801 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:05.080376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:35873)
0927 15:06:05.080638 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:05.080701 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260927 15:06:05.276436 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:05.276030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:60117)
0927 15:06:05.276294 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:05.276347 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260927 15:06:05.975281 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:05.974809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:56121)
0927 15:06:05.975117 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:05.975179 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":77,"threads_started":1}
I20260927 15:06:06.082293 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:06.081832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:38619)
0927 15:06:06.082107 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:06.082195 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260927 15:06:06.215039 17356 raft_consensus.cc:482] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:06.215161 17356 raft_consensus.cc:504] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:06:06.215646 17356 leader_election.cc:288] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343), 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015)
W20260927 15:06:06.216465 16943 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
W20260927 15:06:06.216604 16941 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
I20260927 15:06:06.216696 16941 leader_election.cc:302] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbcfd79ba9eb489990091414aca5d91c; no voters: 43743b3a4f20406085872fdfa9025df8, c712fcc4fe7740099001b5baffdf132b
I20260927 15:06:06.216877 17356 raft_consensus.cc:2726] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:06:06.277616 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:06.277208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:51331)
0927 15:06:06.277468 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:06.277526 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260927 15:06:06.976657 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:06.976225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:35575)
0927 15:06:06.976511 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:06.976563 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
I20260927 15:06:07.083667 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:07.083187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:42395)
0927 15:06:07.083508 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:07.083569 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260927 15:06:07.278846 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:07.278433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:48881)
0927 15:06:07.278694 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:07.278752 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260927 15:06:07.978003 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:07.977488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:51875)
0927 15:06:07.977759 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:07.977885 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260927 15:06:08.084908 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:08.084488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:60713)
0927 15:06:08.084748 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:08.084811 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260927 15:06:08.280081 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:08.279605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:36165)
0927 15:06:08.279901 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:08.279969 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":71,"threads_started":1}
I20260927 15:06:08.979221 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:08.978773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:50661)
0927 15:06:08.979059 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:08.979116 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260927 15:06:09.086193 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:09.085706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:60215)
0927 15:06:09.086038 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:09.086098 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":137,"threads_started":1}
I20260927 15:06:09.281387 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:09.280913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:39947)
0927 15:06:09.281176 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:09.281287 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260927 15:06:09.980436 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:09.980019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:43055)
0927 15:06:09.980278 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:09.980340 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260927 15:06:10.087354 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:10.086928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:60027)
0927 15:06:10.087204 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:10.087258 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260927 15:06:10.282606 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:10.282183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:39351)
0927 15:06:10.282461 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:10.282515 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":72,"threads_started":1}
I20260927 15:06:10.606307 17370 raft_consensus.cc:482] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:10.606436 17370 raft_consensus.cc:504] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c712fcc4fe7740099001b5baffdf132b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45343 } } peers { permanent_uuid: "43743b3a4f20406085872fdfa9025df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42015 } } peers { permanent_uuid: "cbcfd79ba9eb489990091414aca5d91c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } }
I20260927 15:06:10.606915 17370 leader_election.cc:288] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343), 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015)
W20260927 15:06:10.607669 16941 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43743b3a4f20406085872fdfa9025df8 (127.0.98.3:42015): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
W20260927 15:06:10.607758 16943 leader_election.cc:341] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c712fcc4fe7740099001b5baffdf132b (127.0.98.2:45343): Not found: Tablet not found: a4e4d4fea5644217bedac9544eeb63c4
I20260927 15:06:10.607883 16943 leader_election.cc:302] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbcfd79ba9eb489990091414aca5d91c; no voters: 43743b3a4f20406085872fdfa9025df8, c712fcc4fe7740099001b5baffdf132b
I20260927 15:06:10.608040 17370 raft_consensus.cc:2726] T a4e4d4fea5644217bedac9544eeb63c4 P cbcfd79ba9eb489990091414aca5d91c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:06:10.981600 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:10.981198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:44511)
0927 15:06:10.981447 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:10.981507 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260927 15:06:11.088835 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:11.088279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:43511)
0927 15:06:11.088624 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:11.088704 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260927 15:06:11.283995 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:11.283502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:50119)
0927 15:06:11.283807 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:11.283888 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260927 15:06:11.983023 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:11.982588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:39863)
0927 15:06:11.982858 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:11.982924 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260927 15:06:12.090277 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:12.089867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:51775)
0927 15:06:12.090121 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:12.090180 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260927 15:06:12.279224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 16884
I20260927 15:06:12.285503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 17057
I20260927 15:06:12.285661 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:12.285102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:37569)
0927 15:06:12.285460 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:12.285533 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":73,"threads_started":1}
I20260927 15:06:12.291464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 17191
2026-09-27T15:06:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59282 ms)
[----------] 30 tests from SecurityITest (243830 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:12 dist-test-slave-ml7t krb5kdc[17379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:12 dist-test-slave-ml7t krb5kdc[17379](info): set up 2 sockets
Sep 27 15:06:12 dist-test-slave-ml7t krb5kdc[17379](info): commencing operation
krb5kdc: starting...
W20260927 15:06:13.401926 392 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:13 dist-test-slave-ml7t krb5kdc[17379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:13.524507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42759
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:13.633656 17392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:13.633926 17392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:13.634014 17392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:13.637969 17392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:13.638029 17392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:13.638048 17392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:13.638065 17392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:13.638080 17392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:13.643355 17392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:13.644449 17392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:13.645506 17392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:06:13.654433 17392 server_base.cc:1034] running on GCE node
W20260927 15:06:13.654510 17397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:13.654693 17398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:13.654737 17400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:13.655359 17392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:13.656461 17392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:13.657675 17392 hybrid_clock.cc:648] HybridClock initialized: now 1790521573657643 us; error 44 us; skew 500 ppm
Sep 27 15:06:13 dist-test-slave-ml7t krb5kdc[17379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:13.660468 17392 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:13.661384 17392 webserver.cc:460] Webserver started at http://127.0.98.62:38857/ using document root <none> and password file <none>
I20260927 15:06:13.662021 17392 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:13.662086 17392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:13.662300 17392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:13.664295 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "724d86ac38c249d1b8a161b980bba0d4"
format_stamp: "Formatted at 2026-09-27 15:06:13 on dist-test-slave-ml7t"
server_key: "d5f75af97f5baef57526e0984c289056"
server_key_iv: "ef3abcbffd7ecd11c76ca7436132e9d4"
server_key_version: "encryptionkey@0"
I20260927 15:06:13.664855 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "724d86ac38c249d1b8a161b980bba0d4"
format_stamp: "Formatted at 2026-09-27 15:06:13 on dist-test-slave-ml7t"
server_key: "d5f75af97f5baef57526e0984c289056"
server_key_iv: "ef3abcbffd7ecd11c76ca7436132e9d4"
server_key_version: "encryptionkey@0"
I20260927 15:06:13.668596 17392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:06:13.671114 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:13.672364 17392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:13.672485 17392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "724d86ac38c249d1b8a161b980bba0d4"
format_stamp: "Formatted at 2026-09-27 15:06:13 on dist-test-slave-ml7t"
server_key: "d5f75af97f5baef57526e0984c289056"
server_key_iv: "ef3abcbffd7ecd11c76ca7436132e9d4"
server_key_version: "encryptionkey@0"
I20260927 15:06:13.672591 17392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:13.692107 17392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:13.695295 17392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:13.695503 17392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:13.702683 17392 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42759
I20260927 15:06:13.702709 17459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42759 every 8 connection(s)
I20260927 15:06:13.703658 17392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:13.706842 17460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:13.711179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 17392
I20260927 15:06:13.711334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:13.711660 392 external_mini_cluster.cc:1428] Setting key ffdd70d3557184df5f0ccab26602ba7c
I20260927 15:06:13.713325 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4: Bootstrap starting.
I20260927 15:06:13.715778 17460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:13.716559 17460 log.cc:824] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:13.718883 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4: No bootstrap required, opened a new log
Sep 27 15:06:13 dist-test-slave-ml7t krb5kdc[17379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:13.723220 17460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724d86ac38c249d1b8a161b980bba0d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260927 15:06:13.723479 17460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:13.723567 17460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724d86ac38c249d1b8a161b980bba0d4, State: Initialized, Role: FOLLOWER
I20260927 15:06:13.724005 17460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724d86ac38c249d1b8a161b980bba0d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260927 15:06:13.724151 17460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:13.724252 17460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:13.724378 17460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:13.725435 17460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724d86ac38c249d1b8a161b980bba0d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260927 15:06:13.725823 17460 leader_election.cc:302] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 724d86ac38c249d1b8a161b980bba0d4; no voters:
I20260927 15:06:13.726166 17460 leader_election.cc:288] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:13.726233 17465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:13.726424 17465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [term 1 LEADER]: Becoming Leader. State: Replica: 724d86ac38c249d1b8a161b980bba0d4, State: Running, Role: LEADER
I20260927 15:06:13.726761 17465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724d86ac38c249d1b8a161b980bba0d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260927 15:06:13.727188 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:13.713331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.0.98.62:42759)
0927 15:06:13.713873 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:13.713895 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:13.713942 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:06:13.714604 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:13.714620 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:13.714992 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:13.715341 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:13.716257 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:13.717504 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:13.718503 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:13.719016 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:13.719170 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:13.722068 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:13.722073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:13.722086 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:13.722113 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:13.724092 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:13.724618 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:13.724622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:13.724627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:13.724736 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:13.725015 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:13.725017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:13.725018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:13.725440 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:13.725628 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:13.725944 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 15:06:13.726275 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":174,"threads_started":1}
I20260927 15:06:13.727442 17460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:13.728376 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724d86ac38c249d1b8a161b980bba0d4. Latest consensus state: current_term: 1 leader_uuid: "724d86ac38c249d1b8a161b980bba0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724d86ac38c249d1b8a161b980bba0d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } } }
I20260927 15:06:13.728490 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:13.728816 17467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "724d86ac38c249d1b8a161b980bba0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724d86ac38c249d1b8a161b980bba0d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } } }
I20260927 15:06:13.728893 17467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:13.729434 17474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:13.732546 17474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:13.738281 17474 catalog_manager.cc:1347] Generated new cluster ID: 0e0a324a1002421ba05c3c77402e2575
I20260927 15:06:13.738365 17474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:13.745332 17474 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:13.746351 17474 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:13.753286 17474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4: Generated new TSK 0
I20260927 15:06:13.754136 17474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:06:13.787034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:13.905284 17486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:13.905586 17486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:13.905630 17486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:13.909857 17486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:13.909925 17486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:13.909986 17486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:06:13.915334 17486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:13.916415 17486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:13.917361 17486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:13.926751 17491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:13.926749 17492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:13.926738 17494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:13.927094 17486 server_base.cc:1034] running on GCE node
I20260927 15:06:13.927495 17486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:13.928166 17486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:13.929374 17486 hybrid_clock.cc:648] HybridClock initialized: now 1790521573929360 us; error 26 us; skew 500 ppm
Sep 27 15:06:13 dist-test-slave-ml7t krb5kdc[17379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:13.932088 17486 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:06:13.933073 17486 webserver.cc:460] Webserver started at http://127.0.98.1:40319/ using document root <none> and password file <none>
I20260927 15:06:13.933626 17486 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:13.933714 17486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:13.933962 17486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:13.935952 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ee80334d50d499e95f600e908ef6a42"
format_stamp: "Formatted at 2026-09-27 15:06:13 on dist-test-slave-ml7t"
server_key: "05e6e15d12007cece9afc4da1882f918"
server_key_iv: "0d89b25086dcff97091142be1e83317b"
server_key_version: "encryptionkey@0"
I20260927 15:06:13.936523 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ee80334d50d499e95f600e908ef6a42"
format_stamp: "Formatted at 2026-09-27 15:06:13 on dist-test-slave-ml7t"
server_key: "05e6e15d12007cece9afc4da1882f918"
server_key_iv: "0d89b25086dcff97091142be1e83317b"
server_key_version: "encryptionkey@0"
I20260927 15:06:13.940259 17486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:06:13.942889 17501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:13.944218 17486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:13.944418 17486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "8ee80334d50d499e95f600e908ef6a42"
format_stamp: "Formatted at 2026-09-27 15:06:13 on dist-test-slave-ml7t"
server_key: "05e6e15d12007cece9afc4da1882f918"
server_key_iv: "0d89b25086dcff97091142be1e83317b"
server_key_version: "encryptionkey@0"
I20260927 15:06:13.944541 17486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:13.956146 17486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:13.959276 17486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:13.959497 17486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:13.960130 17486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:13.961024 17486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:13.961098 17486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:13.961179 17486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:13.961225 17486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:13.970669 17486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42003
I20260927 15:06:13.970712 17614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42003 every 8 connection(s)
I20260927 15:06:13.971612 17486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:13.973762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 17486
I20260927 15:06:13.973975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:13.974269 392 external_mini_cluster.cc:1428] Setting key 2fcccb77382a56c6c385eef032a8d332
Sep 27 15:06:13 dist-test-slave-ml7t krb5kdc[17379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:06:13.987826 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:13.973394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43829 (local address 127.0.98.62:42759)
0927 15:06:13.973612 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:13.973618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:13.974525 (+ 907us) server_negotiation.cc:407] Connection header received
0927 15:06:13.976106 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:13.976109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:13.976164 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:13.976257 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:13.977648 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:13.978362 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:13.980224 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:13.980480 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:13.980572 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:13.983610 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:13.983613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:13.983615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:13.983641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:13.985239 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:13.985883 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:13.985886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:13.985887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:13.985972 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:13.986353 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:13.986356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:13.986357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:13.986537 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:13.986617 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:13.987403 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 15:06:13.987560 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 15:06:13.988442 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:13.973819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42759 (local address 127.0.98.1:43829)
0927 15:06:13.974355 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:13.974408 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:13.975904 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:13.976455 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:13.976462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:13.976955 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:13.977489 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:13.977504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:13.978515 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:13.978520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:13.980093 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:13.980107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:13.980650 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:13.980656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:13.980888 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:13.981535 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:13.981552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:13.983419 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:13.985379 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:13.985388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:13.985405 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:13.985671 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:13.986060 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:13.986062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:13.986064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:13.986247 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:13.986633 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:13.986638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:13.987024 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 15:06:13.987248 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 15:06:13.987557 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":176,"threads_started":1}
I20260927 15:06:13.989696 17615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42759
I20260927 15:06:13.990007 17615 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:13.990512 17615 heartbeater.cc:511] Master 127.0.98.62:42759 requested a full tablet report, sending...
I20260927 15:06:13.992256 17424 ts_manager.cc:194] Registered new tserver with Master: 8ee80334d50d499e95f600e908ef6a42 (127.0.98.1:42003)
I20260927 15:06:13.994056 17424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:06:14.006202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:14.115084 17620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:14.115386 17620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:14.115471 17620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:14.119464 17620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:14.119565 17620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:14.119663 17620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:06:14.125197 17620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:14.126395 17620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:14.127488 17620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:14.137351 17626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:14.137379 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:14.137416 17620 server_base.cc:1034] running on GCE node
W20260927 15:06:14.137565 17628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:14.137989 17620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:14.138659 17620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:14.139863 17620 hybrid_clock.cc:648] HybridClock initialized: now 1790521574139837 us; error 37 us; skew 500 ppm
Sep 27 15:06:14 dist-test-slave-ml7t krb5kdc[17379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:14.142685 17620 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:06:14.143604 17620 webserver.cc:460] Webserver started at http://127.0.98.2:35955/ using document root <none> and password file <none>
I20260927 15:06:14.144416 17620 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:14.144503 17620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:14.144758 17620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:14.147590 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "3476318f11084cd19ba399b459c80624"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "6edc6863b821c34b992f8c0f9398c0d9"
server_key_iv: "ff9113b9d44d00720c8fa53d32b16b73"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.148161 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3476318f11084cd19ba399b459c80624"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "6edc6863b821c34b992f8c0f9398c0d9"
server_key_iv: "ff9113b9d44d00720c8fa53d32b16b73"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.152033 17620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:06:14.154637 17635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:14.155907 17620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:14.156052 17620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "3476318f11084cd19ba399b459c80624"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "6edc6863b821c34b992f8c0f9398c0d9"
server_key_iv: "ff9113b9d44d00720c8fa53d32b16b73"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.156167 17620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:14.165997 17620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:14.169135 17620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:14.169344 17620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:14.169965 17620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:14.170852 17620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:14.170938 17620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:14.171026 17620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:14.171063 17620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:14.180289 17620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37523
I20260927 15:06:14.180317 17748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37523 every 8 connection(s)
I20260927 15:06:14.181201 17620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:14.182606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 17620
I20260927 15:06:14.182734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:14.183079 392 external_mini_cluster.cc:1428] Setting key 44f64249920be961b305a625b9b2eaf3
Sep 27 15:06:14 dist-test-slave-ml7t krb5kdc[17379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:06:14.196494 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.183099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51113 (local address 127.0.98.62:42759)
0927 15:06:14.183253 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.183257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.184314 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 15:06:14.186228 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.186232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.186274 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.186374 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:14.187570 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.188480 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.189719 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.190181 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.190310 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.192706 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:14.192711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:14.192713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:14.192748 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:14.194337 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:14.194849 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:14.194852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:14.194853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:14.194926 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:14.195223 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:14.195225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:14.195226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:14.195400 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:14.195518 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.196075 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.196222 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:06:14.197079 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.183611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42759 (local address 127.0.98.2:51113)
0927 15:06:14.184145 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.184201 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:14.185983 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:14.186568 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:14.186580 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:14.186950 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:14.187424 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.187437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.188608 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.188611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.189589 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.189598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.190354 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.190357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.190491 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.191116 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:14.191132 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:14.192545 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:14.194449 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:14.194454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:14.194464 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:14.194714 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:14.195012 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:14.195015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:14.195016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:14.195144 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:14.195522 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:14.195527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:14.195825 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 15:06:14.195983 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:06:14.196233 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260927 15:06:14.198393 17749 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42759
I20260927 15:06:14.198622 17749 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:14.199098 17749 heartbeater.cc:511] Master 127.0.98.62:42759 requested a full tablet report, sending...
I20260927 15:06:14.200224 17424 ts_manager.cc:194] Registered new tserver with Master: 3476318f11084cd19ba399b459c80624 (127.0.98.2:37523)
I20260927 15:06:14.201119 17424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:06:14.212390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:14.321111 17754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:14.321374 17754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:14.321419 17754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:14.325392 17754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:14.325444 17754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:14.325500 17754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:06:14.330783 17754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:14.331790 17754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:14.332774 17754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:14.342504 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:14.342610 17754 server_base.cc:1034] running on GCE node
W20260927 15:06:14.342504 17759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:14.342523 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:14.343245 17754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:14.343966 17754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:14.345153 17754 hybrid_clock.cc:648] HybridClock initialized: now 1790521574345137 us; error 28 us; skew 500 ppm
Sep 27 15:06:14 dist-test-slave-ml7t krb5kdc[17379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:14.348218 17754 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:06:14.349232 17754 webserver.cc:460] Webserver started at http://127.0.98.3:38913/ using document root <none> and password file <none>
I20260927 15:06:14.349998 17754 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:14.350128 17754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:14.350359 17754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:14.352416 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ffb2b8263c340f8908d8bc6a095201c"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "8aacf7943bf3d41600d223091e184ba9"
server_key_iv: "20917344c3b0629ea25cf408df3fac35"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.352959 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ffb2b8263c340f8908d8bc6a095201c"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "8aacf7943bf3d41600d223091e184ba9"
server_key_iv: "20917344c3b0629ea25cf408df3fac35"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.356634 17754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:06:14.359133 17769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:14.360409 17754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:14.360550 17754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "3ffb2b8263c340f8908d8bc6a095201c"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "8aacf7943bf3d41600d223091e184ba9"
server_key_iv: "20917344c3b0629ea25cf408df3fac35"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.360658 17754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:14.373402 17754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:14.376356 17754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:14.376564 17754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:14.377126 17754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:14.377974 17754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:14.378039 17754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:14.378105 17754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:14.378165 17754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:14.387326 17754 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43543
I20260927 15:06:14.387349 17882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43543 every 8 connection(s)
I20260927 15:06:14.388191 17754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 15:06:14 dist-test-slave-ml7t krb5kdc[17379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:14.398530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 17754
I20260927 15:06:14.398669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:14.398970 392 external_mini_cluster.cc:1428] Setting key a086ddbe11d9fe3c2af8092334326183
I20260927 15:06:14.401850 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.389916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38105 (local address 127.0.98.62:42759)
0927 15:06:14.390077 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.390081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.390946 (+ 865us) server_negotiation.cc:407] Connection header received
0927 15:06:14.392156 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.392159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.392205 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.392280 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:14.393426 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.394023 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.395142 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.395347 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.395425 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.397884 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:14.397887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:14.397889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:14.397914 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:14.399526 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:14.400197 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:14.400200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:14.400201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:14.400303 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:14.400726 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:14.400728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:14.400729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:14.400904 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:14.401017 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.401573 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.401668 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:06:14.402714 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.390341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42759 (local address 127.0.98.3:38105)
0927 15:06:14.390777 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.390830 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:14.391947 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:14.392469 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:14.392475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:14.392829 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:14.393283 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.393296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.394140 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.394144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.395032 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.395042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.395484 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.395487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.395621 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.396249 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:14.396265 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:14.397635 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:14.399663 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:14.399669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:14.399683 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:14.400070 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:14.400447 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:14.400450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:14.400452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:14.400625 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:14.401039 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:14.401046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:14.401362 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 15:06:14.401525 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:06:14.401831 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260927 15:06:14.403915 17883 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42759
I20260927 15:06:14.404117 17883 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:14.404543 17883 heartbeater.cc:511] Master 127.0.98.62:42759 requested a full tablet report, sending...
I20260927 15:06:14.405495 17424 ts_manager.cc:194] Registered new tserver with Master: 3ffb2b8263c340f8908d8bc6a095201c (127.0.98.3:43543)
I20260927 15:06:14.406131 17424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38105
I20260927 15:06:14.412599 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:14.421681 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.414497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.0.98.62:42759)
0927 15:06:14.414609 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.414613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.414777 (+ 164us) server_negotiation.cc:407] Connection header received
0927 15:06:14.414893 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.414895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.414930 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.414997 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:14.415676 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.416172 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.417030 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.417232 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.417311 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.418803 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:14.418806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:14.418808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:14.418829 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:14.420300 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:14.420674 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:14.420677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:14.420678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:14.420748 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:14.420972 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:14.420974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:14.420975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:14.421167 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:14.421311 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.421474 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.421566 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:06:14.424556 17424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:14.428221 17424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:14.443377 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.437873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.0.98.1:42003)
0927 15:06:14.438221 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.438234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.438257 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:06:14.438340 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.438347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.438562 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.439059 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:14.439210 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.440388 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.441839 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.442659 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.442778 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.442954 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.443097 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.443201 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
I20260927 15:06:14.443404 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.436972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37523 (local address 127.0.0.1:44998)
0927 15:06:14.437600 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.437651 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:14.437835 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:14.439274 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:14.439277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:14.439295 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:14.439370 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.439376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.440306 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.440310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.441896 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.441907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.442939 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.442943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.443067 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.443091 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:06:14.443142 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 15:06:14.443229 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":99,"spinlock_wait_cycles":2432,"thread_start_us":98,"threads_started":1}
I20260927 15:06:14.443748 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.437215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.0.98.2:37523)
0927 15:06:14.438783 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.438794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.438826 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:06:14.438885 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.438889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.439079 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.439267 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:14.439477 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.440177 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.442026 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.442797 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.443321 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.443456 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.443515 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.443613 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":159,"threads_started":1}
I20260927 15:06:14.443761 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.438180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.0.98.3:43543)
0927 15:06:14.438595 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.438607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.438633 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:06:14.438710 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.438714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.438900 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.439056 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:14.439514 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.440598 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.442041 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.442698 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.442833 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.442974 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.443476 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.443584 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
I20260927 15:06:14.443802 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.437206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:58888)
0927 15:06:14.437636 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.437651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:14.437833 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:14.438749 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:14.438756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:14.438779 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:14.438919 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.438926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.440515 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.440518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.441672 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.441681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.442796 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.442803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.442905 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.442942 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:06:14.443057 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:06:14.443212 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":46,"spinlock_wait_cycles":47104,"thread_start_us":142,"threads_started":1}
I20260927 15:06:14.443404 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.437745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43543 (local address 127.0.0.1:50816)
0927 15:06:14.438428 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.438441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:14.438527 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:14.439040 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:14.439043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:14.439058 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:14.439161 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.439170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.440713 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.440718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.441921 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.441929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.443035 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.443040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.443123 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.443141 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:06:14.443204 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:06:14.443260 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":18,"thread_start_us":104,"threads_started":1}
I20260927 15:06:14.446619 17817 tablet_service.cc:1467] Processing CreateTablet for tablet dc455d07c7b048f88a484f823263a077 (DEFAULT_TABLE table=test-table [id=82673af620d042bdba828c07551fec8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:14.446761 17683 tablet_service.cc:1467] Processing CreateTablet for tablet dc455d07c7b048f88a484f823263a077 (DEFAULT_TABLE table=test-table [id=82673af620d042bdba828c07551fec8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:14.447623 17817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc455d07c7b048f88a484f823263a077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:14.447760 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc455d07c7b048f88a484f823263a077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:14.447500 17549 tablet_service.cc:1467] Processing CreateTablet for tablet dc455d07c7b048f88a484f823263a077 (DEFAULT_TABLE table=test-table [id=82673af620d042bdba828c07551fec8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:14.448400 17549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc455d07c7b048f88a484f823263a077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:14.456269 17900 tablet_bootstrap.cc:492] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42: Bootstrap starting.
I20260927 15:06:14.457028 17902 tablet_bootstrap.cc:492] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Bootstrap starting.
I20260927 15:06:14.458696 17900 tablet_bootstrap.cc:654] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:14.459107 17901 tablet_bootstrap.cc:492] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Bootstrap starting.
I20260927 15:06:14.459332 17902 tablet_bootstrap.cc:654] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:14.460693 17902 log.cc:824] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:14.461308 17901 tablet_bootstrap.cc:654] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:14.462668 17901 log.cc:824] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:14.462790 17902 tablet_bootstrap.cc:492] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: No bootstrap required, opened a new log
I20260927 15:06:14.462987 17902 ts_tablet_manager.cc:1397] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:06:14.463008 17900 log.cc:824] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:14.464692 17901 tablet_bootstrap.cc:492] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: No bootstrap required, opened a new log
I20260927 15:06:14.464844 17900 tablet_bootstrap.cc:492] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42: No bootstrap required, opened a new log
I20260927 15:06:14.464953 17901 ts_tablet_manager.cc:1397] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:14.465072 17900 ts_tablet_manager.cc:1397] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 15:06:14.467257 17902 raft_consensus.cc:348] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.467669 17902 raft_consensus.cc:374] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:14.467729 17902 raft_consensus.cc:729] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffb2b8263c340f8908d8bc6a095201c, State: Initialized, Role: FOLLOWER
I20260927 15:06:14.468218 17902 consensus_queue.cc:260] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.468238 17900 raft_consensus.cc:348] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.468482 17900 raft_consensus.cc:374] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:14.468545 17900 raft_consensus.cc:729] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee80334d50d499e95f600e908ef6a42, State: Initialized, Role: FOLLOWER
I20260927 15:06:14.468883 17901 raft_consensus.cc:348] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.469007 17900 consensus_queue.cc:260] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.469111 17901 raft_consensus.cc:374] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:14.469158 17901 raft_consensus.cc:729] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3476318f11084cd19ba399b459c80624, State: Initialized, Role: FOLLOWER
I20260927 15:06:14.469333 17902 ts_tablet_manager.cc:1428] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:14.469585 17901 consensus_queue.cc:260] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.469691 17883 heartbeater.cc:503] Master 127.0.98.62:42759 was elected leader, sending a full tablet report...
I20260927 15:06:14.470031 17900 ts_tablet_manager.cc:1428] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:14.470315 17749 heartbeater.cc:503] Master 127.0.98.62:42759 was elected leader, sending a full tablet report...
I20260927 15:06:14.470492 17901 ts_tablet_manager.cc:1428] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:06:14.470789 17615 heartbeater.cc:503] Master 127.0.98.62:42759 was elected leader, sending a full tablet report...
W20260927 15:06:14.472740 17616 tablet.cc:2369] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:06:14.639130 17884 tablet.cc:2369] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:14.664872 17907 raft_consensus.cc:482] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:14.665117 17907 raft_consensus.cc:504] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.666527 17907 leader_election.cc:288] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3476318f11084cd19ba399b459c80624 (127.0.98.2:37523), 3ffb2b8263c340f8908d8bc6a095201c (127.0.98.3:43543)
I20260927 15:06:14.670722 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.667156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32893 (local address 127.0.98.3:43543)
0927 15:06:14.667368 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.667373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.667388 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:06:14.667511 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.667513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.667583 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.667661 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:14.667968 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.668477 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.669624 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.670091 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.670203 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.670277 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.670505 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.670586 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:06:14.670763 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.666871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58265 (local address 127.0.98.2:37523)
0927 15:06:14.667039 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.667044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.667140 (+ 96us) server_negotiation.cc:407] Connection header received
0927 15:06:14.667410 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.667415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.667477 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.667560 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:14.667882 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.668457 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.669574 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.670102 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.670234 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.670296 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.670573 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.670653 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:06:14.670751 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.666722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37523 (local address 127.0.98.1:58265)
0927 15:06:14.667044 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.667060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:14.667264 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:14.667589 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:14.667591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:14.667611 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:14.667756 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.667763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.668569 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.668576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.669448 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.669457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.670249 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.670253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.670422 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.670440 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:06:14.670539 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 15:06:14.670603 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":86,"thread_start_us":130,"threads_started":1}
I20260927 15:06:14.670763 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.666837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43543 (local address 127.0.98.1:32893)
0927 15:06:14.667248 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.667270 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:14.667399 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:14.667693 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:14.667696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:14.667720 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:14.667845 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.667852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.668621 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.668624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.669479 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:14.669488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.670220 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.670224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:14.670347 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.670403 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 15:06:14.670467 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:06:14.670555 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":3,"thread_start_us":135,"threads_started":1}
I20260927 15:06:14.671669 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc455d07c7b048f88a484f823263a077" candidate_uuid: "8ee80334d50d499e95f600e908ef6a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffb2b8263c340f8908d8bc6a095201c" is_pre_election: true
I20260927 15:06:14.671680 17703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc455d07c7b048f88a484f823263a077" candidate_uuid: "8ee80334d50d499e95f600e908ef6a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3476318f11084cd19ba399b459c80624" is_pre_election: true
I20260927 15:06:14.672019 17703 raft_consensus.cc:2454] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee80334d50d499e95f600e908ef6a42 in term 0.
I20260927 15:06:14.672066 17837 raft_consensus.cc:2454] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee80334d50d499e95f600e908ef6a42 in term 0.
I20260927 15:06:14.672614 17505 leader_election.cc:302] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3476318f11084cd19ba399b459c80624, 3ffb2b8263c340f8908d8bc6a095201c, 8ee80334d50d499e95f600e908ef6a42; no voters:
I20260927 15:06:14.672914 17907 raft_consensus.cc:2781] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:14.673017 17907 raft_consensus.cc:482] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:14.673080 17907 raft_consensus.cc:3037] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:14.674103 17907 raft_consensus.cc:504] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.674482 17907 leader_election.cc:288] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [CANDIDATE]: Term 1 election: Requested vote from peers 3476318f11084cd19ba399b459c80624 (127.0.98.2:37523), 3ffb2b8263c340f8908d8bc6a095201c (127.0.98.3:43543)
I20260927 15:06:14.674829 17703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc455d07c7b048f88a484f823263a077" candidate_uuid: "8ee80334d50d499e95f600e908ef6a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3476318f11084cd19ba399b459c80624"
I20260927 15:06:14.674868 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc455d07c7b048f88a484f823263a077" candidate_uuid: "8ee80334d50d499e95f600e908ef6a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffb2b8263c340f8908d8bc6a095201c"
I20260927 15:06:14.674927 17703 raft_consensus.cc:3037] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:14.675020 17837 raft_consensus.cc:3037] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:14.676131 17703 raft_consensus.cc:2454] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee80334d50d499e95f600e908ef6a42 in term 1.
I20260927 15:06:14.676366 17837 raft_consensus.cc:2454] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee80334d50d499e95f600e908ef6a42 in term 1.
I20260927 15:06:14.676687 17503 leader_election.cc:302] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffb2b8263c340f8908d8bc6a095201c, 8ee80334d50d499e95f600e908ef6a42; no voters:
I20260927 15:06:14.676868 17907 raft_consensus.cc:2781] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:14.677071 17907 raft_consensus.cc:686] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee80334d50d499e95f600e908ef6a42, State: Running, Role: LEADER
I20260927 15:06:14.677433 17907 consensus_queue.cc:237] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } }
I20260927 15:06:14.680492 17424 catalog_manager.cc:5495] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee80334d50d499e95f600e908ef6a42 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8ee80334d50d499e95f600e908ef6a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee80334d50d499e95f600e908ef6a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 } health_report { overall_health: UNKNOWN } } }
W20260927 15:06:14.682420 17750 tablet.cc:2369] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:14.737934 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.734422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.0.98.1:42003)
0927 15:06:14.734574 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.734578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.734595 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:06:14.734708 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.734710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.734762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.734858 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:14.735061 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.735551 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.736415 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.736655 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:14.736739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:14.737017 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:14.737435 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:14.737530 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:14.737643 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 15:06:14.737689 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:06:14.743835 17837 raft_consensus.cc:1261] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 1 FOLLOWER]: Refusing update from remote peer 8ee80334d50d499e95f600e908ef6a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:14.743847 17703 raft_consensus.cc:1261] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 1 FOLLOWER]: Refusing update from remote peer 8ee80334d50d499e95f600e908ef6a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:14.744520 17907 consensus_queue.cc:1035] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3476318f11084cd19ba399b459c80624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:14.744745 17911 consensus_queue.cc:1035] T dc455d07c7b048f88a484f823263a077 P 8ee80334d50d499e95f600e908ef6a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffb2b8263c340f8908d8bc6a095201c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:14.752833 17915 mvcc.cc:205] Tried to move back new op lower bound from 7333976370143219712 to 7333976369881538560. Current Snapshot: MvccSnapshot[applied={T|T < 7333976370143219712}]
I20260927 15:06:14.754251 17916 mvcc.cc:205] Tried to move back new op lower bound from 7333976370143219712 to 7333976369881538560. Current Snapshot: MvccSnapshot[applied={T|T < 7333976370143219712}]
I20260927 15:06:14.760885 17424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:14.761109 17424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:14.763777 17424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 724d86ac38c249d1b8a161b980bba0d4: Sending DeleteTablet for 3 replicas of tablet dc455d07c7b048f88a484f823263a077
I20260927 15:06:14.764619 17549 tablet_service.cc:1514] Processing DeleteTablet for tablet dc455d07c7b048f88a484f823263a077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58888
I20260927 15:06:14.764752 17683 tablet_service.cc:1514] Processing DeleteTablet for tablet dc455d07c7b048f88a484f823263a077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44998
I20260927 15:06:14.765446 17817 tablet_service.cc:1514] Processing DeleteTablet for tablet dc455d07c7b048f88a484f823263a077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50816
I20260927 15:06:14.765887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 17486
I20260927 15:06:14.766841 17925 tablet_replica.cc:330] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: stopping tablet replica
I20260927 15:06:14.767204 17925 raft_consensus.cc:2229] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:14.767550 17925 raft_consensus.cc:2258] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:14.767799 17924 tablet_replica.cc:330] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: stopping tablet replica
I20260927 15:06:14.768141 17924 raft_consensus.cc:2229] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:14.768452 17924 raft_consensus.cc:2258] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:14.769014 17925 ts_tablet_manager.cc:1905] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:14.769835 17924 ts_tablet_manager.cc:1905] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:14.771309 17925 ts_tablet_manager.cc:1918] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:14.771417 17925 log.cc:1196] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/dc455d07c7b048f88a484f823263a077
I20260927 15:06:14.771706 17925 ts_tablet_manager.cc:1939] T dc455d07c7b048f88a484f823263a077 P 3ffb2b8263c340f8908d8bc6a095201c: Deleting consensus metadata
I20260927 15:06:14.772831 17409 catalog_manager.cc:4841] TS 3ffb2b8263c340f8908d8bc6a095201c (127.0.98.3:43543): tablet dc455d07c7b048f88a484f823263a077 (table test-table [id=82673af620d042bdba828c07551fec8c]) successfully deleted
I20260927 15:06:14.773183 17924 ts_tablet_manager.cc:1918] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:14.773286 17924 log.cc:1196] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/dc455d07c7b048f88a484f823263a077
I20260927 15:06:14.773609 17924 ts_tablet_manager.cc:1939] T dc455d07c7b048f88a484f823263a077 P 3476318f11084cd19ba399b459c80624: Deleting consensus metadata
I20260927 15:06:14.774571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 17620
W20260927 15:06:14.774570 17408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42003 (error 108)
I20260927 15:06:14.775763 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.775511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:58904)
0927 15:06:14.775634 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.775691 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:06:14.775946 17408 catalog_manager.cc:4584] TS 8ee80334d50d499e95f600e908ef6a42 (127.0.98.1:42003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc455d07c7b048f88a484f823263a077: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
I20260927 15:06:14.781453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 17754
I20260927 15:06:14.782649 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.781699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37523 (local address 127.0.0.1:45014)
0927 15:06:14.782466 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:14.782535 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663}
I20260927 15:06:14.787182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 17392
2026-09-27T15:06:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2495 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T15:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:14Z Disabled control of system clock
I20260927 15:06:14.818163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45007
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:06:14.928278 17930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:14.928546 17930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:14.928581 17930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:14.932616 17930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:14.932680 17930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:14.932695 17930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:14.932713 17930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:14.932762 17930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:14.937561 17930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:14.938694 17930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:14.940171 17930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:14.948915 17936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:14.949083 17935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:14.949178 17930 server_base.cc:1034] running on GCE node
W20260927 15:06:14.949215 17938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:14.949926 17930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:14.951100 17930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:14.952319 17930 hybrid_clock.cc:648] HybridClock initialized: now 1790521574952319 us; error 55 us; skew 500 ppm
I20260927 15:06:14.954409 17930 webserver.cc:460] Webserver started at http://127.0.98.62:36243/ using document root <none> and password file <none>
I20260927 15:06:14.955013 17930 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:14.955114 17930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:14.955365 17930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:14.957541 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "8a041c25837c4a2c9ea86453bb35423c"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "7cde34b348d5dd01e9fd6993abeb2f25"
server_key_iv: "178382c35613cd85a5c488d51cab5b77"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.958201 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a041c25837c4a2c9ea86453bb35423c"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "7cde34b348d5dd01e9fd6993abeb2f25"
server_key_iv: "178382c35613cd85a5c488d51cab5b77"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.962057 17930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:06:14.964622 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:14.965905 17930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:14.966056 17930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "8a041c25837c4a2c9ea86453bb35423c"
format_stamp: "Formatted at 2026-09-27 15:06:14 on dist-test-slave-ml7t"
server_key: "7cde34b348d5dd01e9fd6993abeb2f25"
server_key_iv: "178382c35613cd85a5c488d51cab5b77"
server_key_version: "encryptionkey@0"
I20260927 15:06:14.966171 17930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:14.980362 17930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:14.981020 17930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:14.981210 17930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:14.988314 17996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45007 every 8 connection(s)
I20260927 15:06:14.988312 17930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45007
I20260927 15:06:14.989262 17930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:14.992296 17997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:14.994086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 17930
I20260927 15:06:14.994221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:14.994522 392 external_mini_cluster.cc:1428] Setting key 56f41e9962fff72bc3d743b981c1050f
I20260927 15:06:14.999464 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c: Bootstrap starting.
I20260927 15:06:15.001950 17997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:15.002807 17997 log.cc:824] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:15.004779 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:14.996006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.0.98.62:45007)
0927 15:06:14.996643 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:14.996659 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:14.996696 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:06:14.997378 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:14.997395 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:14.997863 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:14.998156 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:14.998553 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:14.999511 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.000434 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.000913 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.001085 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.002444 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:15.002449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:15.002471 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:15.003063 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:15.003133 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:15.003156 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:15.003281 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.003473 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.003797 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":138,"threads_started":1}
I20260927 15:06:15.005313 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c: No bootstrap required, opened a new log
I20260927 15:06:15.009843 17997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a041c25837c4a2c9ea86453bb35423c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45007 } }
I20260927 15:06:15.010151 17997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:15.010190 17997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a041c25837c4a2c9ea86453bb35423c, State: Initialized, Role: FOLLOWER
I20260927 15:06:15.010681 17997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a041c25837c4a2c9ea86453bb35423c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45007 } }
I20260927 15:06:15.010818 17997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:15.010892 17997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:15.010977 17997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:15.012092 17997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a041c25837c4a2c9ea86453bb35423c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45007 } }
I20260927 15:06:15.012467 17997 leader_election.cc:302] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a041c25837c4a2c9ea86453bb35423c; no voters:
I20260927 15:06:15.012794 17997 leader_election.cc:288] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:15.013034 18002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:15.013283 18002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [term 1 LEADER]: Becoming Leader. State: Replica: 8a041c25837c4a2c9ea86453bb35423c, State: Running, Role: LEADER
I20260927 15:06:15.013689 18002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a041c25837c4a2c9ea86453bb35423c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45007 } }
I20260927 15:06:15.014386 17997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:15.015558 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a041c25837c4a2c9ea86453bb35423c. Latest consensus state: current_term: 1 leader_uuid: "8a041c25837c4a2c9ea86453bb35423c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a041c25837c4a2c9ea86453bb35423c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45007 } } }
I20260927 15:06:15.015723 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:15.015537 18003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a041c25837c4a2c9ea86453bb35423c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a041c25837c4a2c9ea86453bb35423c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45007 } } }
I20260927 15:06:15.016005 18003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:15.016491 18011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:15.019855 18011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:15.025796 18011 catalog_manager.cc:1347] Generated new cluster ID: 2b7f4a0607d548d9b5e31d047b71fb68
I20260927 15:06:15.025879 18011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:15.055322 18011 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:15.056540 18011 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:15.071584 18011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c: Generated new TSK 0
I20260927 15:06:15.072366 18011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:15.083551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45007
--builtin_ntp_servers=127.0.98.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:06:15.229077 18021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:15.229348 18021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:15.229387 18021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:15.233480 18021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:15.233549 18021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:15.233605 18021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:06:15.238215 18021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:15.239184 18021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:15.240461 18021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:15.249832 18026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:15.249882 18027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:15.250087 18021 server_base.cc:1034] running on GCE node
W20260927 15:06:15.249912 18029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:15.250664 18021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:15.251370 18021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:15.252559 18021 hybrid_clock.cc:648] HybridClock initialized: now 1790521575252526 us; error 48 us; skew 500 ppm
I20260927 15:06:15.254532 18021 webserver.cc:460] Webserver started at http://127.0.98.1:40567/ using document root <none> and password file <none>
I20260927 15:06:15.255137 18021 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:15.255234 18021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:15.255478 18021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:15.257650 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "46d7e8517aa74dc3a1739012297739e7"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "1e263959eef77cbcd8a2a4d8fde3221b"
server_key_iv: "f552628efaf69e94b614906bdecdb2ae"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.258258 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "46d7e8517aa74dc3a1739012297739e7"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "1e263959eef77cbcd8a2a4d8fde3221b"
server_key_iv: "f552628efaf69e94b614906bdecdb2ae"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.261817 18021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:15.264201 18035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.265394 18021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:06:15.265529 18021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "46d7e8517aa74dc3a1739012297739e7"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "1e263959eef77cbcd8a2a4d8fde3221b"
server_key_iv: "f552628efaf69e94b614906bdecdb2ae"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.265637 18021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:15.279090 18021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:15.279755 18021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:15.279954 18021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:15.280524 18021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:15.281445 18021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:15.281515 18021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.281586 18021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:15.281630 18021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.290505 18021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46759
I20260927 15:06:15.290539 18148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46759 every 8 connection(s)
I20260927 15:06:15.291385 18021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:15.300901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 18021
I20260927 15:06:15.301033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:15.301090 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.293197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53355 (local address 127.0.98.62:45007)
0927 15:06:15.293394 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.293400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.294513 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 15:06:15.295898 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.295901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.295951 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.296029 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:15.296947 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.297930 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.299191 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.299414 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.299497 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.300569 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:15.300573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:15.300575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:15.300586 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:15.300603 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:15.300610 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:15.300668 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.300857 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.300909 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 15:06:15.301347 392 external_mini_cluster.cc:1428] Setting key 340c1373c4dd5696f2888ef2d7c90831
I20260927 15:06:15.302141 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.293639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45007 (local address 127.0.98.1:53355)
0927 15:06:15.294355 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.294398 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.295701 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.296201 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.296209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.296332 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:15.296815 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.296828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.298063 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.298069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.299075 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.299086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.299577 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.299580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.299723 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.300317 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:15.300319 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:15.300356 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:15.300416 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:15.300697 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:15.300708 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:15.300714 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.300852 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.301153 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":155,"threads_started":1}
I20260927 15:06:15.303349 18149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45007
I20260927 15:06:15.303478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45007
--builtin_ntp_servers=127.0.98.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:06:15.303581 18149 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:15.304145 18149 heartbeater.cc:511] Master 127.0.98.62:45007 requested a full tablet report, sending...
I20260927 15:06:15.305579 17961 ts_manager.cc:194] Registered new tserver with Master: 46d7e8517aa74dc3a1739012297739e7 (127.0.98.1:46759)
I20260927 15:06:15.306918 17961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53355
W20260927 15:06:15.414845 18152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:15.415145 18152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:15.415190 18152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:15.419430 18152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:15.419481 18152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:15.419536 18152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:06:15.424307 18152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:15.425411 18152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:15.426808 18152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:15.436632 18158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:15.436672 18157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:15.436632 18160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:15.436960 18152 server_base.cc:1034] running on GCE node
I20260927 15:06:15.437565 18152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:15.438318 18152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:15.439528 18152 hybrid_clock.cc:648] HybridClock initialized: now 1790521575439491 us; error 53 us; skew 500 ppm
I20260927 15:06:15.441566 18152 webserver.cc:460] Webserver started at http://127.0.98.2:38115/ using document root <none> and password file <none>
I20260927 15:06:15.442206 18152 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:15.442268 18152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:15.442509 18152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:15.444558 18152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa515afca06a4abebf74242b4a00d3cf"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "796a637effa474b39e83afe9909f3e57"
server_key_iv: "560602bdf4b5d3d9405703712cec7a1d"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.445117 18152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa515afca06a4abebf74242b4a00d3cf"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "796a637effa474b39e83afe9909f3e57"
server_key_iv: "560602bdf4b5d3d9405703712cec7a1d"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.448709 18152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:06:15.451198 18166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.452535 18152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:15.452695 18152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "aa515afca06a4abebf74242b4a00d3cf"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "796a637effa474b39e83afe9909f3e57"
server_key_iv: "560602bdf4b5d3d9405703712cec7a1d"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.452831 18152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:15.476799 18152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:15.477591 18152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:15.477896 18152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:15.478560 18152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:15.479678 18152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:15.479763 18152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.479843 18152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:15.479904 18152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.489324 18152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36717
I20260927 15:06:15.489375 18279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36717 every 8 connection(s)
I20260927 15:06:15.490446 18152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:15.500020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 18152
I20260927 15:06:15.500187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:15.500497 392 external_mini_cluster.cc:1428] Setting key 53404954d58e5e99b4a985c3bab5147d
I20260927 15:06:15.500519 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.492409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40625 (local address 127.0.98.62:45007)
0927 15:06:15.492554 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.492558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.493667 (+ 1109us) server_negotiation.cc:407] Connection header received
0927 15:06:15.495235 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.495239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.495313 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.495403 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:15.496400 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.497000 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.498298 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.498569 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.498696 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.499915 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:15.499919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:15.499923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:15.499938 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:15.499970 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:15.499979 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:15.500063 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.500321 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.500394 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:06:15.502085 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.492761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45007 (local address 127.0.98.2:40625)
0927 15:06:15.493504 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.493547 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.494992 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.495591 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.495598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.495693 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:15.496227 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.496252 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.497144 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.497148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.498151 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.498162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.498745 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.498749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.498925 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.499629 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:15.499631 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:15.499674 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:15.499742 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:15.500078 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:15.500090 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:15.500098 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.500693 (+ 595us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.501009 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":104,"threads_started":1}
I20260927 15:06:15.502795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45007
--builtin_ntp_servers=127.0.98.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:06:15.503362 18280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45007
I20260927 15:06:15.503607 18280 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:15.504227 18280 heartbeater.cc:511] Master 127.0.98.62:45007 requested a full tablet report, sending...
I20260927 15:06:15.505327 17961 ts_manager.cc:194] Registered new tserver with Master: aa515afca06a4abebf74242b4a00d3cf (127.0.98.2:36717)
I20260927 15:06:15.505946 17961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40625
W20260927 15:06:15.617890 18283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:15.618163 18283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:15.618199 18283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:15.622156 18283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:15.622201 18283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:15.622283 18283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:06:15.626962 18283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:15.628129 18283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:15.629554 18283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:15.638586 18288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:15.638665 18291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:15.638779 18283 server_base.cc:1034] running on GCE node
W20260927 15:06:15.638587 18289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:15.639308 18283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:15.639940 18283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:15.641142 18283 hybrid_clock.cc:648] HybridClock initialized: now 1790521575641128 us; error 25 us; skew 500 ppm
I20260927 15:06:15.642999 18283 webserver.cc:460] Webserver started at http://127.0.98.3:44901/ using document root <none> and password file <none>
I20260927 15:06:15.643575 18283 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:15.643674 18283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:15.643905 18283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:15.646001 18283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "5ca6d1e41cf305844244c42c107da64a"
server_key_iv: "a649721b75ec8aac3857111a1b798d13"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.646552 18283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "5ca6d1e41cf305844244c42c107da64a"
server_key_iv: "a649721b75ec8aac3857111a1b798d13"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.650154 18283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:15.652546 18297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.653825 18283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:15.653951 18283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5"
format_stamp: "Formatted at 2026-09-27 15:06:15 on dist-test-slave-ml7t"
server_key: "5ca6d1e41cf305844244c42c107da64a"
server_key_iv: "a649721b75ec8aac3857111a1b798d13"
server_key_version: "encryptionkey@0"
I20260927 15:06:15.654060 18283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:15.696149 18283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:15.696929 18283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:15.697142 18283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:15.697795 18283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:15.698798 18283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:15.698866 18283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.698937 18283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:15.698988 18283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:15.708302 18283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42705
I20260927 15:06:15.708333 18410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42705 every 8 connection(s)
I20260927 15:06:15.709226 18283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:15.710170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 18283
I20260927 15:06:15.710280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:15.710568 392 external_mini_cluster.cc:1428] Setting key 768cfbce36d92fae686eee063a578c60
I20260927 15:06:15.719293 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.711485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38107 (local address 127.0.98.62:45007)
0927 15:06:15.711604 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.711608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.713009 (+ 1401us) server_negotiation.cc:407] Connection header received
0927 15:06:15.714522 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.714525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.714577 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.714657 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:15.715641 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.716178 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.717309 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.717540 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.717653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.718812 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:15.718816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:15.718819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:15.718837 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:15.718855 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:15.718863 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:15.718938 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.719098 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.719153 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:06:15.720077 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.712075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45007 (local address 127.0.98.3:38107)
0927 15:06:15.712872 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.712927 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.714329 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.714844 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.714855 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.714946 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:15.715506 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.715519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.716316 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.716323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.717177 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.717185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.717686 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.717689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.717903 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.718526 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:15.718527 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:15.718565 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:15.718623 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:15.718952 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:15.718960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:15.718966 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.719096 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.719367 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260927 15:06:15.721226 18411 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45007
I20260927 15:06:15.721441 18411 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:15.721952 18411 heartbeater.cc:511] Master 127.0.98.62:45007 requested a full tablet report, sending...
I20260927 15:06:15.722872 17961 ts_manager.cc:194] Registered new tserver with Master: 1232ad9e86fe40ecaa17b8ac92cc19b5 (127.0.98.3:42705)
I20260927 15:06:15.723515 17961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38107
I20260927 15:06:15.724763 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:15.730296 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.726672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.0.98.62:45007)
0927 15:06:15.726838 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.726843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.726983 (+ 140us) server_negotiation.cc:407] Connection header received
0927 15:06:15.727075 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.727077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.727116 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.727177 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:15.727407 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.727948 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.728713 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.728943 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.729023 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.729888 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:15.729891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:15.729894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:15.729904 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:15.729921 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:15.729929 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:15.729989 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.730093 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.730137 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:06:15.733309 17961 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:15.736565 17961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:15.751684 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.746146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.0.98.2:36717)
0927 15:06:15.746843 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.746857 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.746881 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:06:15.746955 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.746960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.747151 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.747605 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:15.747746 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.748847 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.750386 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.750977 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.751129 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.751359 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.751416 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.751508 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":142,"threads_started":1}
I20260927 15:06:15.751680 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.745612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36717 (local address 127.0.0.1:56378)
0927 15:06:15.746179 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.746195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.746278 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.747329 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.747333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.747352 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:15.747451 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.747458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.748971 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.748974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.750253 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.750264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.751128 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.751133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.751243 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.751282 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.751391 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.751508 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":59,"thread_start_us":114,"threads_started":1}
I20260927 15:06:15.751794 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.745390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42705 (local address 127.0.0.1:34420)
0927 15:06:15.745707 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.745754 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.745942 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.746831 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.746839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.746866 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:15.747020 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.747028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.748661 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.748665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.750276 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.750291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.751367 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.751371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.751468 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.751489 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.751539 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.751607 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":158,"thread_start_us":80,"threads_started":1}
I20260927 15:06:15.752415 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.746132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46759 (local address 127.0.0.1:59640)
0927 15:06:15.746461 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.746475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.746551 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.747716 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.747719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.747737 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:15.747819 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.747824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.748803 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.748809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.750346 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.750359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.751367 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.751371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.752125 (+ 754us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.752148 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.752221 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.752290 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":977,"thread_start_us":96,"threads_started":1}
I20260927 15:06:15.752568 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.746560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.98.1:46759)
0927 15:06:15.747352 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.747364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.747389 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:06:15.747465 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.747468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.747589 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.747742 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:15.747914 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.748673 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.750479 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.751218 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.751419 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.751604 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.752283 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.752395 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":95,"threads_started":1}
I20260927 15:06:15.753217 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.745497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.98.3:42705)
0927 15:06:15.746242 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.746259 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.746290 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:06:15.746379 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.746384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.746610 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.747190 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:15.747354 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.748530 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.750437 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.751218 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.752706 (+ 1488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.752892 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.752945 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.753048 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":184,"threads_started":1}
I20260927 15:06:15.754760 18214 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0655c08b0f45b2b4b5bdacebda935a (DEFAULT_TABLE table=test-table [id=ab7794aa90fa477fba9c8e2e0df9ad57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:15.755812 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0655c08b0f45b2b4b5bdacebda935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:15.756388 18345 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0655c08b0f45b2b4b5bdacebda935a (DEFAULT_TABLE table=test-table [id=ab7794aa90fa477fba9c8e2e0df9ad57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:15.756389 18083 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0655c08b0f45b2b4b5bdacebda935a (DEFAULT_TABLE table=test-table [id=ab7794aa90fa477fba9c8e2e0df9ad57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:15.757454 18083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0655c08b0f45b2b4b5bdacebda935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:15.757694 18345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0655c08b0f45b2b4b5bdacebda935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:15.762677 18428 tablet_bootstrap.cc:492] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Bootstrap starting.
I20260927 15:06:15.764433 18430 tablet_bootstrap.cc:492] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Bootstrap starting.
I20260927 15:06:15.765221 18428 tablet_bootstrap.cc:654] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:15.765267 18429 tablet_bootstrap.cc:492] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7: Bootstrap starting.
I20260927 15:06:15.766197 18428 log.cc:824] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:15.766742 18430 tablet_bootstrap.cc:654] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:15.767208 18429 tablet_bootstrap.cc:654] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:15.767629 18430 log.cc:824] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:15.767953 18429 log.cc:824] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:15.768427 18428 tablet_bootstrap.cc:492] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: No bootstrap required, opened a new log
I20260927 15:06:15.768636 18428 ts_tablet_manager.cc:1397] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:15.769863 18430 tablet_bootstrap.cc:492] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: No bootstrap required, opened a new log
I20260927 15:06:15.770054 18430 ts_tablet_manager.cc:1397] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:15.771332 18429 tablet_bootstrap.cc:492] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7: No bootstrap required, opened a new log
I20260927 15:06:15.771544 18429 ts_tablet_manager.cc:1397] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 15:06:15.773264 18428 raft_consensus.cc:348] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.773576 18428 raft_consensus.cc:374] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:15.773669 18428 raft_consensus.cc:729] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa515afca06a4abebf74242b4a00d3cf, State: Initialized, Role: FOLLOWER
I20260927 15:06:15.774206 18428 consensus_queue.cc:260] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.775007 18280 heartbeater.cc:503] Master 127.0.98.62:45007 was elected leader, sending a full tablet report...
I20260927 15:06:15.774904 18430 raft_consensus.cc:348] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.775156 18430 raft_consensus.cc:374] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:15.775208 18430 raft_consensus.cc:729] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1232ad9e86fe40ecaa17b8ac92cc19b5, State: Initialized, Role: FOLLOWER
I20260927 15:06:15.775346 18428 ts_tablet_manager.cc:1428] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 15:06:15.775648 18430 consensus_queue.cc:260] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.775998 18429 raft_consensus.cc:348] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.776278 18429 raft_consensus.cc:374] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:15.776329 18411 heartbeater.cc:503] Master 127.0.98.62:45007 was elected leader, sending a full tablet report...
I20260927 15:06:15.776338 18429 raft_consensus.cc:729] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d7e8517aa74dc3a1739012297739e7, State: Initialized, Role: FOLLOWER
I20260927 15:06:15.776674 18430 ts_tablet_manager.cc:1428] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 15:06:15.776782 18429 consensus_queue.cc:260] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.777490 18149 heartbeater.cc:503] Master 127.0.98.62:45007 was elected leader, sending a full tablet report...
I20260927 15:06:15.777797 18429 ts_tablet_manager.cc:1428] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
W20260927 15:06:15.792686 18150 tablet.cc:2369] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:15.948717 18434 raft_consensus.cc:482] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:15.948918 18434 raft_consensus.cc:504] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.950127 18434 leader_election.cc:288] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1232ad9e86fe40ecaa17b8ac92cc19b5 (127.0.98.3:42705), 46d7e8517aa74dc3a1739012297739e7 (127.0.98.1:46759)
I20260927 15:06:15.953847 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.950406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42705 (local address 127.0.98.2:54989)
0927 15:06:15.950537 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.950553 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.950657 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.951003 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.951007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.951033 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:15.951148 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.951157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.951932 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.951939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.952739 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.952748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.953411 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.953414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.953501 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.953528 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.953612 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.953689 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260927 15:06:15.953847 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.950507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54989 (local address 127.0.98.3:42705)
0927 15:06:15.950661 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.950666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.950684 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:06:15.950749 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.950755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.950813 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.950924 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:15.951283 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.951811 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.952876 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.953280 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.953409 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.953470 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.953638 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.953711 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:06:15.954061 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.950698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46759 (local address 127.0.98.2:51453)
0927 15:06:15.951005 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:15.951017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:15.951086 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:15.951317 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:15.951320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:15.951339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:15.951455 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.951461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.952193 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.952198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.952955 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:15.952963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.953614 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.953617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:15.953715 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.953732 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:06:15.953853 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 15:06:15.953924 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260927 15:06:15.954047 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.950964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51453 (local address 127.0.98.1:46759)
0927 15:06:15.951102 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.951106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.951118 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:06:15.951163 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.951164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.951200 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.951286 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:15.951584 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.952086 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.953080 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.953477 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.953589 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.953646 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.953871 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.953937 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:06:15.954574 18365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0655c08b0f45b2b4b5bdacebda935a" candidate_uuid: "aa515afca06a4abebf74242b4a00d3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" is_pre_election: true
I20260927 15:06:15.954715 18103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0655c08b0f45b2b4b5bdacebda935a" candidate_uuid: "aa515afca06a4abebf74242b4a00d3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46d7e8517aa74dc3a1739012297739e7" is_pre_election: true
I20260927 15:06:15.954905 18365 raft_consensus.cc:2454] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa515afca06a4abebf74242b4a00d3cf in term 0.
I20260927 15:06:15.955044 18103 raft_consensus.cc:2454] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa515afca06a4abebf74242b4a00d3cf in term 0.
I20260927 15:06:15.955404 18169 leader_election.cc:302] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1232ad9e86fe40ecaa17b8ac92cc19b5, aa515afca06a4abebf74242b4a00d3cf; no voters:
I20260927 15:06:15.955693 18434 raft_consensus.cc:2781] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:15.955796 18434 raft_consensus.cc:482] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:15.955852 18434 raft_consensus.cc:3037] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:15.956936 18434 raft_consensus.cc:504] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
I20260927 15:06:15.957293 18434 leader_election.cc:288] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [CANDIDATE]: Term 1 election: Requested vote from peers 1232ad9e86fe40ecaa17b8ac92cc19b5 (127.0.98.3:42705), 46d7e8517aa74dc3a1739012297739e7 (127.0.98.1:46759)
I20260927 15:06:15.957662 18365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0655c08b0f45b2b4b5bdacebda935a" candidate_uuid: "aa515afca06a4abebf74242b4a00d3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5"
I20260927 15:06:15.957690 18103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0655c08b0f45b2b4b5bdacebda935a" candidate_uuid: "aa515afca06a4abebf74242b4a00d3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46d7e8517aa74dc3a1739012297739e7"
I20260927 15:06:15.957852 18365 raft_consensus.cc:3037] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:15.957862 18103 raft_consensus.cc:3037] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:15.958945 18365 raft_consensus.cc:2454] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa515afca06a4abebf74242b4a00d3cf in term 1.
I20260927 15:06:15.959030 18103 raft_consensus.cc:2454] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa515afca06a4abebf74242b4a00d3cf in term 1.
I20260927 15:06:15.959287 18169 leader_election.cc:302] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1232ad9e86fe40ecaa17b8ac92cc19b5, aa515afca06a4abebf74242b4a00d3cf; no voters:
I20260927 15:06:15.959498 18434 raft_consensus.cc:2781] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:15.959714 18434 raft_consensus.cc:686] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 1 LEADER]: Becoming Leader. State: Replica: aa515afca06a4abebf74242b4a00d3cf, State: Running, Role: LEADER
I20260927 15:06:15.960068 18434 consensus_queue.cc:237] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } }
W20260927 15:06:15.960212 18412 tablet.cc:2369] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:15.963393 17960 catalog_manager.cc:5495] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf reported cstate change: term changed from 0 to 1, leader changed from <none> to aa515afca06a4abebf74242b4a00d3cf (127.0.98.2). New cstate: current_term: 1 leader_uuid: "aa515afca06a4abebf74242b4a00d3cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa515afca06a4abebf74242b4a00d3cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36717 } health_report { overall_health: HEALTHY } } }
W20260927 15:06:15.991662 18281 tablet.cc:2369] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:15.994993 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:15.990803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.0.98.2:36717)
0927 15:06:15.990969 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:15.990975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:15.990994 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:06:15.991083 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:15.991086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:15.991157 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:15.991287 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:15.991540 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.992335 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.993191 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:15.993575 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:15.993698 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:15.994063 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:15.994572 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:15.994694 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:15.994777 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 15:06:15.994833 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:06:16.001289 18365 raft_consensus.cc:1261] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 1 FOLLOWER]: Refusing update from remote peer aa515afca06a4abebf74242b4a00d3cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:16.001364 18103 raft_consensus.cc:1261] T 6a0655c08b0f45b2b4b5bdacebda935a P 46d7e8517aa74dc3a1739012297739e7 [term 1 FOLLOWER]: Refusing update from remote peer aa515afca06a4abebf74242b4a00d3cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:16.002069 18434 consensus_queue.cc:1035] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1232ad9e86fe40ecaa17b8ac92cc19b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:16.002300 18438 consensus_queue.cc:1035] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "46d7e8517aa74dc3a1739012297739e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:16.011173 18442 mvcc.cc:205] Tried to move back new op lower bound from 7333976375293804544 to 7333976375135207424. Current Snapshot: MvccSnapshot[applied={T|T < 7333976375293804544}]
I20260927 15:06:16.019681 17961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:16.019886 17961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:16.022475 17961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a041c25837c4a2c9ea86453bb35423c: Sending DeleteTablet for 3 replicas of tablet 6a0655c08b0f45b2b4b5bdacebda935a
I20260927 15:06:16.023428 18214 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0655c08b0f45b2b4b5bdacebda935a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:16 UTC) from {username='slave'} at 127.0.0.1:56378
I20260927 15:06:16.023914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 18021
I20260927 15:06:16.023985 18345 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0655c08b0f45b2b4b5bdacebda935a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:16 UTC) from {username='slave'} at 127.0.0.1:34420
I20260927 15:06:16.026369 18451 tablet_replica.cc:330] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: stopping tablet replica
I20260927 15:06:16.026736 18451 raft_consensus.cc:2229] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:16.027182 18451 raft_consensus.cc:2258] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:16.028721 18451 ts_tablet_manager.cc:1905] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:06:16.030776 17947 connection.cc:535] client connection to 127.0.98.1:46759 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 15:06:16.030956 17947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:06:16.031829 18451 ts_tablet_manager.cc:1918] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:16.031934 18451 log.cc:1196] T 6a0655c08b0f45b2b4b5bdacebda935a P aa515afca06a4abebf74242b4a00d3cf: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/6a0655c08b0f45b2b4b5bdacebda935a
I20260927 15:06:16.032015 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:16.031668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46759 (local address 127.0.0.1:59656)
0927 15:06:16.031812 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:16.031891 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:06:16.032078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 18152
W20260927 15:06:16.032605 17947 catalog_manager.cc:4584] TS 46d7e8517aa74dc3a1739012297739e7 (127.0.98.1:46759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a0655c08b0f45b2b4b5bdacebda935a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46759: connect: Connection refused (error 111)
I20260927 15:06:16.033928 18452 tablet_replica.cc:330] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: stopping tablet replica
I20260927 15:06:16.034284 18452 raft_consensus.cc:2229] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:16.034638 18452 raft_consensus.cc:2258] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:16.036072 18452 ts_tablet_manager.cc:1905] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:16.038431 18452 ts_tablet_manager.cc:1918] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:16.038533 18452 log.cc:1196] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/6a0655c08b0f45b2b4b5bdacebda935a
I20260927 15:06:16.038904 18452 ts_tablet_manager.cc:1939] T 6a0655c08b0f45b2b4b5bdacebda935a P 1232ad9e86fe40ecaa17b8ac92cc19b5: Deleting consensus metadata
I20260927 15:06:16.039134 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:16.038803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36717 (local address 127.0.0.1:56400)
0927 15:06:16.038961 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:16.039036 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 15:06:16.039166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 18283
I20260927 15:06:16.044655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 17930
2026-09-27T15:06:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1257 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:16 dist-test-slave-ml7t krb5kdc[18456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:16 dist-test-slave-ml7t krb5kdc[18456](info): set up 2 sockets
Sep 27 15:06:16 dist-test-slave-ml7t krb5kdc[18456](info): commencing operation
krb5kdc: starting...
W20260927 15:06:17.082217 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:17 dist-test-slave-ml7t krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:17.204530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:17.316303 18469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:17.316653 18469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:17.316743 18469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:17.320797 18469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:17.320897 18469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:17.320942 18469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:17.320999 18469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:17.321040 18469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:17.326576 18469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:17.327653 18469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:17.328672 18469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:17.337373 18477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:17.337455 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:17.337584 18469 server_base.cc:1034] running on GCE node
W20260927 15:06:17.337352 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:17.338243 18469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:17.339362 18469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:17.340550 18469 hybrid_clock.cc:648] HybridClock initialized: now 1790521577340521 us; error 41 us; skew 500 ppm
Sep 27 15:06:17 dist-test-slave-ml7t krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:17.343305 18469 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:17.344172 18469 webserver.cc:460] Webserver started at http://127.0.98.62:36651/ using document root <none> and password file <none>
I20260927 15:06:17.344722 18469 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:17.344818 18469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:17.345046 18469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:17.347102 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "dc9e374e13da4c2e8f74cf65c90924eb"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "d3dcc7db8977c90094ab2fec2b7c123f"
server_key_iv: "80c3eb36798f085ec0d6bcfba1d48ad1"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.347656 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc9e374e13da4c2e8f74cf65c90924eb"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "d3dcc7db8977c90094ab2fec2b7c123f"
server_key_iv: "80c3eb36798f085ec0d6bcfba1d48ad1"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.351432 18469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 15:06:17.354143 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:17.355476 18469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:06:17.355644 18469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "dc9e374e13da4c2e8f74cf65c90924eb"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "d3dcc7db8977c90094ab2fec2b7c123f"
server_key_iv: "80c3eb36798f085ec0d6bcfba1d48ad1"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.355770 18469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:17.374795 18469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:17.378136 18469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:17.378387 18469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:17.385958 18469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39361
I20260927 15:06:17.385990 18536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39361 every 8 connection(s)
I20260927 15:06:17.386961 18469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:17.389619 18537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:17.390993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 18469
I20260927 15:06:17.391144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:17.391423 392 external_mini_cluster.cc:1428] Setting key f9f6edf1a35de32abe8105c601563815
I20260927 15:06:17.396091 18537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb: Bootstrap starting.
I20260927 15:06:17.398661 18537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:17.399536 18537 log.cc:824] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:06:17 dist-test-slave-ml7t krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:17.401441 18537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb: No bootstrap required, opened a new log
I20260927 15:06:17.406193 18537 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39361 } }
I20260927 15:06:17.406462 18537 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:17.406531 18537 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9e374e13da4c2e8f74cf65c90924eb, State: Initialized, Role: FOLLOWER
I20260927 15:06:17.406991 18537 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39361 } }
I20260927 15:06:17.407089 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:17.392894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.0.98.62:39361)
0927 15:06:17.393324 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:17.393346 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:17.393390 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:06:17.394271 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:17.394296 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:17.394698 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:17.395068 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:17.395810 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.397247 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.398201 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.398936 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.399091 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:17.401757 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:17.401763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:17.401803 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:17.401838 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:17.403970 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:17.404431 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:17.404437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:17.404443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:17.404582 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:17.404854 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:17.404857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:17.404859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:17.405361 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:17.405578 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:17.405813 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 15:06:17.406110 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
I20260927 15:06:17.407156 18537 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:17.407212 18537 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:17.407302 18537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:17.408419 18537 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39361 } }
I20260927 15:06:17.408758 18537 leader_election.cc:302] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc9e374e13da4c2e8f74cf65c90924eb; no voters:
I20260927 15:06:17.409075 18537 leader_election.cc:288] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:17.409190 18542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:17.409425 18542 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [term 1 LEADER]: Becoming Leader. State: Replica: dc9e374e13da4c2e8f74cf65c90924eb, State: Running, Role: LEADER
I20260927 15:06:17.409741 18542 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39361 } }
I20260927 15:06:17.410346 18537 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:17.411501 18544 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9e374e13da4c2e8f74cf65c90924eb. Latest consensus state: current_term: 1 leader_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39361 } } }
I20260927 15:06:17.411650 18544 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:17.411947 18543 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9e374e13da4c2e8f74cf65c90924eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39361 } } }
I20260927 15:06:17.412073 18543 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:17.412060 18551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:17.415525 18551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:17.421079 18551 catalog_manager.cc:1347] Generated new cluster ID: bf36bd64a2164abd9400916fea16bcd3
I20260927 15:06:17.421150 18551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:17.432708 18551 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:17.433646 18551 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:17.449620 18551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb: Generated new TSK 0
I20260927 15:06:17.450412 18551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:06:17.485306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39361
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:17.623152 18563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:17.623425 18563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:17.623468 18563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:17.627473 18563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:17.627530 18563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:17.627588 18563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:06:17.632961 18563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:17.634004 18563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:17.634971 18563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:17.644404 18571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:17.644402 18568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:17.644397 18569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:17.644691 18563 server_base.cc:1034] running on GCE node
I20260927 15:06:17.645093 18563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:17.645748 18563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:17.647011 18563 hybrid_clock.cc:648] HybridClock initialized: now 1790521577646980 us; error 53 us; skew 500 ppm
Sep 27 15:06:17 dist-test-slave-ml7t krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:17.650049 18563 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:06:17.651008 18563 webserver.cc:460] Webserver started at http://127.0.98.1:46183/ using document root <none> and password file <none>
I20260927 15:06:17.651602 18563 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:17.651695 18563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:17.651926 18563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:17.654086 18563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "c515f34b390240fea969f22552de6864"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "d70465d22602ea084fc3fbd7042ff2e7"
server_key_iv: "0667379fa72891b8a7e9576b7292b569"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.654660 18563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c515f34b390240fea969f22552de6864"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "d70465d22602ea084fc3fbd7042ff2e7"
server_key_iv: "0667379fa72891b8a7e9576b7292b569"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.658386 18563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:06:17.660949 18578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:17.662393 18563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:17.662523 18563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "c515f34b390240fea969f22552de6864"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "d70465d22602ea084fc3fbd7042ff2e7"
server_key_iv: "0667379fa72891b8a7e9576b7292b569"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.662636 18563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:17.689204 18563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:17.692701 18563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:17.692917 18563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:17.693543 18563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:17.694432 18563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:17.694504 18563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:17.694577 18563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:17.694620 18563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:17.703886 18563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33281
I20260927 15:06:17.703939 18691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33281 every 8 connection(s)
I20260927 15:06:17.704808 18563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:17.712946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 18563
I20260927 15:06:17.713049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:17.713384 392 external_mini_cluster.cc:1428] Setting key fd2e4ff80c28c02265e9d1fd2e05d8cd
Sep 27 15:06:17 dist-test-slave-ml7t krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:17.718356 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:17.706780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49143 (local address 127.0.98.62:39361)
0927 15:06:17.706928 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:17.706933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:17.707848 (+ 915us) server_negotiation.cc:407] Connection header received
0927 15:06:17.708707 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:17.708710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:17.708762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:17.708839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:17.710079 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.710595 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.711799 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.712030 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.712164 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:17.714733 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:17.714737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:17.714744 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:17.714768 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:17.716196 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:17.716724 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:17.716729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:17.716731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:17.716811 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:17.717210 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:17.717214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:17.717215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:17.717404 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:17.717512 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:17.718067 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 15:06:17.718188 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:06:17.719197 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:17.707235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39361 (local address 127.0.98.1:49143)
0927 15:06:17.707683 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:17.707727 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:17.708508 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:17.708994 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:17.709000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:17.709335 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:17.709941 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:17.709957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.710707 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.710710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:17.711695 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:17.711704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.712181 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.712184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:17.712316 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:17.712941 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:17.712955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:17.714567 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:17.716339 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:17.716345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:17.716361 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:17.716620 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:17.716904 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:17.716908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:17.716910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:17.717053 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:17.717510 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:17.717517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:17.717816 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 15:06:17.718027 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 15:06:17.718361 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260927 15:06:17.720520 18692 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39361
I20260927 15:06:17.720789 18692 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:17.721277 18692 heartbeater.cc:511] Master 127.0.98.62:39361 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:06:17.722882 18501 ts_manager.cc:194] Registered new tserver with Master: c515f34b390240fea969f22552de6864 (127.0.98.1:33281)
I20260927 15:06:17.724021 18501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:06:17.740115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39361
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:17.852151 18697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:17.852433 18697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:17.852478 18697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:17.856989 18697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:17.857060 18697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:17.857158 18697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:06:17.862958 18697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:17.864084 18697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:17.865136 18697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:17.875794 18702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:17.875841 18697 server_base.cc:1034] running on GCE node
W20260927 15:06:17.875864 18703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:17.875794 18705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:17.876551 18697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:17.877270 18697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:17.878469 18697 hybrid_clock.cc:648] HybridClock initialized: now 1790521577878444 us; error 49 us; skew 500 ppm
Sep 27 15:06:17 dist-test-slave-ml7t krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:17.881678 18697 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:06:17.882660 18697 webserver.cc:460] Webserver started at http://127.0.98.2:42757/ using document root <none> and password file <none>
I20260927 15:06:17.883299 18697 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:17.883401 18697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:17.883635 18697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:17.885723 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "3af71d798ea041e4805c9ede9e8894e4"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "dea8a436ff76756f3018c2da62c3e67d"
server_key_iv: "97e651db1d93124652eeee3c97ec2569"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.886359 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3af71d798ea041e4805c9ede9e8894e4"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "dea8a436ff76756f3018c2da62c3e67d"
server_key_iv: "97e651db1d93124652eeee3c97ec2569"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.890062 18697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:06:17.892477 18712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:17.893724 18697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:17.893883 18697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "3af71d798ea041e4805c9ede9e8894e4"
format_stamp: "Formatted at 2026-09-27 15:06:17 on dist-test-slave-ml7t"
server_key: "dea8a436ff76756f3018c2da62c3e67d"
server_key_iv: "97e651db1d93124652eeee3c97ec2569"
server_key_version: "encryptionkey@0"
I20260927 15:06:17.893991 18697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:17.910347 18697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:17.914192 18697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:17.914407 18697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:17.915025 18697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:17.915903 18697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:17.915973 18697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:17.916036 18697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:17.916085 18697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:17.925216 18697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46263
I20260927 15:06:17.925239 18825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46263 every 8 connection(s)
I20260927 15:06:17.926210 18697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:17.926653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 18697
I20260927 15:06:17.926774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:17.927100 392 external_mini_cluster.cc:1428] Setting key f4828e1cd55c5f451a32e8f048e9cc57
Sep 27 15:06:17 dist-test-slave-ml7t krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:06:17.940663 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:17.928231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48421 (local address 127.0.98.62:39361)
0927 15:06:17.928370 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:17.928374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:17.929507 (+ 1133us) server_negotiation.cc:407] Connection header received
0927 15:06:17.930486 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:17.930490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:17.930542 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:17.930627 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:17.931827 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.932376 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.933541 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.933761 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.933917 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:17.936562 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:17.936565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:17.936567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:17.936613 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:17.938168 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:17.938813 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:17.938819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:17.938825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:17.938903 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:17.939281 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:17.939284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:17.939286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:17.939598 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:17.939704 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:17.940426 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 15:06:17.940537 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:17.942004 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:17.928808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39361 (local address 127.0.98.2:48421)
0927 15:06:17.929365 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:17.929409 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:17.930284 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:17.930823 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:17.930830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:17.931191 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:17.931676 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:17.931689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.932506 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.932511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:17.933405 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:17.933414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:17.933953 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:17.933957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:17.934173 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:17.934837 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:17.934854 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:17.936374 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:17.938344 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:17.938352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:17.938371 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:17.938685 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:17.939005 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:17.939009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:17.939010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:17.939162 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:17.939753 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:17.939760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:17.940107 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 15:06:17.940364 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 15:06:17.940788 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":144,"threads_started":1}
I20260927 15:06:17.943316 18826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39361
I20260927 15:06:17.943598 18826 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:17.944250 18826 heartbeater.cc:511] Master 127.0.98.62:39361 requested a full tablet report, sending...
I20260927 15:06:17.945500 18501 ts_manager.cc:194] Registered new tserver with Master: 3af71d798ea041e4805c9ede9e8894e4 (127.0.98.2:46263)
I20260927 15:06:17.946372 18501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:06:17.968947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39361
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:18.080629 18831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:18.080899 18831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:18.080941 18831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:18.084987 18831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:18.085047 18831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:18.085148 18831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:06:18.090402 18831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:18.091511 18831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:18.092535 18831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:18.101989 18836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:18.102021 18839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:18.101989 18837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:18.102334 18831 server_base.cc:1034] running on GCE node
I20260927 15:06:18.102708 18831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:18.103299 18831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:18.104501 18831 hybrid_clock.cc:648] HybridClock initialized: now 1790521578104487 us; error 25 us; skew 500 ppm
Sep 27 15:06:18 dist-test-slave-ml7t krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:18.107262 18831 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:06:18.108109 18831 webserver.cc:460] Webserver started at http://127.0.98.3:36295/ using document root <none> and password file <none>
I20260927 15:06:18.108613 18831 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:18.108668 18831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:18.108819 18831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:18.110934 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "345a7cf658ef45949912752a335d8ba5"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "a599c98c213b6b5642c4e00fbb1637e0"
server_key_iv: "057967d87daa68412249a80fbd6564ee"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.111524 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "345a7cf658ef45949912752a335d8ba5"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "a599c98c213b6b5642c4e00fbb1637e0"
server_key_iv: "057967d87daa68412249a80fbd6564ee"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.115290 18831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:06:18.117928 18846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:18.119251 18831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:18.119349 18831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "345a7cf658ef45949912752a335d8ba5"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "a599c98c213b6b5642c4e00fbb1637e0"
server_key_iv: "057967d87daa68412249a80fbd6564ee"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.119465 18831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:18.135591 18831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:18.138763 18831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:18.138916 18831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:18.139441 18831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:18.140254 18831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:18.140300 18831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:18.140338 18831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:18.140354 18831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:18.149194 18831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43513
I20260927 15:06:18.149220 18959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43513 every 8 connection(s)
I20260927 15:06:18.150166 18831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:18.155695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 18831
I20260927 15:06:18.155884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:18.156186 392 external_mini_cluster.cc:1428] Setting key 8fb3e3a60b11417c68eeca25913c1dca
Sep 27 15:06:18 dist-test-slave-ml7t krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:18.165540 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.152162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38219 (local address 127.0.98.62:39361)
0927 15:06:18.152334 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.152338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.153155 (+ 817us) server_negotiation.cc:407] Connection header received
0927 15:06:18.154186 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.154190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.154252 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.154370 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:18.155704 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.156394 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.157736 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.158016 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.158143 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.162102 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:18.162109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:18.162113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:18.162146 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:18.163656 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:18.164171 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:18.164173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:18.164174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:18.164245 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:18.164560 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:18.164562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:18.164563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:18.164728 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:18.164801 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.165299 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.165401 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:06:18.166632 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.152524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39361 (local address 127.0.98.3:38219)
0927 15:06:18.152991 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.153035 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:18.153951 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:18.154555 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:18.154563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:18.154938 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:18.155506 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.155529 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.156547 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.156551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.157588 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.157598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.158182 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.158186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.158430 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.159252 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:18.159284 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:18.161826 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:18.163783 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:18.163788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:18.163801 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:18.164077 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:18.164326 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:18.164328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:18.164330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:18.164478 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:18.164817 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:18.164822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:18.165057 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 15:06:18.165246 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 15:06:18.165555 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260927 15:06:18.167831 18960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39361
I20260927 15:06:18.168097 18960 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:18.168710 18960 heartbeater.cc:511] Master 127.0.98.62:39361 requested a full tablet report, sending...
I20260927 15:06:18.169775 18501 ts_manager.cc:194] Registered new tserver with Master: 345a7cf658ef45949912752a335d8ba5 (127.0.98.3:43513)
I20260927 15:06:18.170389 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:18.170459 18501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38219
I20260927 15:06:18.180225 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.172623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38616 (local address 127.0.98.62:39361)
0927 15:06:18.172749 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.172753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.172947 (+ 194us) server_negotiation.cc:407] Connection header received
0927 15:06:18.173052 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.173054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.173092 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.173164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:18.173884 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.174450 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.175280 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.175499 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.175589 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.177032 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:18.177035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:18.177037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:18.177058 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:18.178554 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:18.179063 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:18.179065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:18.179067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:18.179142 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:18.179477 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:18.179480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:18.179482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:18.179677 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:18.179801 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.180037 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.180114 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:06:18.183085 18501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:18.186161 18501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:18.201095 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.194601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43513 (local address 127.0.0.1:48026)
0927 15:06:18.194949 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.195003 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:18.195130 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:18.196313 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:18.196319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:18.196341 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:18.196547 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.196557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.197687 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.197694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.199226 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.199237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.200298 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.200302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.200450 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.200510 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 15:06:18.200588 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:06:18.200715 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":44,"thread_start_us":71,"threads_started":1}
I20260927 15:06:18.201303 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.195307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.0.98.3:43513)
0927 15:06:18.195717 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.195729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.195805 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:06:18.195893 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.195901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.196132 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.196498 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:18.196674 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.197544 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.199356 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.200157 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.200874 (+ 717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.200995 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.201049 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.201137 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":88,"threads_started":1}
I20260927 15:06:18.201634 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.195420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.0.98.1:33281)
0927 15:06:18.195787 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.195800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.195895 (+ 95us) server_negotiation.cc:407] Connection header received
0927 15:06:18.196028 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.196035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.196227 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.196914 (+ 687us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:18.197080 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.198189 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.199608 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.200392 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.200848 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.200996 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.201048 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.201138 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":66,"threads_started":1}
I20260927 15:06:18.201646 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.194965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33281 (local address 127.0.0.1:39928)
0927 15:06:18.195799 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.195812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:18.195916 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:18.196390 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:18.196394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:18.196412 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:18.196754 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.196763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.198290 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.198296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.199474 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.199484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.200523 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.200527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.200607 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.200624 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:06:18.200662 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 15:06:18.200715 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":90,"spinlock_wait_cycles":1408,"thread_start_us":103,"threads_started":1}
I20260927 15:06:18.201958 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.195435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.98.2:46263)
0927 15:06:18.196183 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.196194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.196215 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:06:18.196288 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.196295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.196464 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.196618 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:18.196902 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.198116 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.199552 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.200111 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.200340 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.200624 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.201624 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.201789 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":128,"threads_started":1}
I20260927 15:06:18.202590 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.194692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46263 (local address 127.0.0.1:35398)
0927 15:06:18.195526 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.195539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:18.195618 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:18.196602 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:18.196605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:18.196628 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:18.196762 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.196768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.198240 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.198243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.199448 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.199455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.201379 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.201383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.201478 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.201500 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:06:18.201572 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:06:18.201636 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":67,"spinlock_wait_cycles":2304,"thread_start_us":94,"threads_started":1}
I20260927 15:06:18.204700 18760 tablet_service.cc:1467] Processing CreateTablet for tablet 0270d45d627349c38c07e14cdb4634e0 (DEFAULT_TABLE table=test-table [id=dd2e7813eaf14549a38d5f61b0e78df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:18.204659 18894 tablet_service.cc:1467] Processing CreateTablet for tablet 0270d45d627349c38c07e14cdb4634e0 (DEFAULT_TABLE table=test-table [id=dd2e7813eaf14549a38d5f61b0e78df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:18.204653 18626 tablet_service.cc:1467] Processing CreateTablet for tablet 0270d45d627349c38c07e14cdb4634e0 (DEFAULT_TABLE table=test-table [id=dd2e7813eaf14549a38d5f61b0e78df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:18.205634 18760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0270d45d627349c38c07e14cdb4634e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:18.205644 18894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0270d45d627349c38c07e14cdb4634e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:18.205644 18626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0270d45d627349c38c07e14cdb4634e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:18.212852 18978 tablet_bootstrap.cc:492] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864: Bootstrap starting.
I20260927 15:06:18.212917 18977 tablet_bootstrap.cc:492] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Bootstrap starting.
I20260927 15:06:18.214084 18979 tablet_bootstrap.cc:492] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Bootstrap starting.
I20260927 15:06:18.214751 18977 tablet_bootstrap.cc:654] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:18.214843 18978 tablet_bootstrap.cc:654] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:18.215757 18977 log.cc:824] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:18.215986 18978 log.cc:824] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:18.216140 18979 tablet_bootstrap.cc:654] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:18.217255 18979 log.cc:824] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:18.217692 18977 tablet_bootstrap.cc:492] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: No bootstrap required, opened a new log
I20260927 15:06:18.217887 18978 tablet_bootstrap.cc:492] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864: No bootstrap required, opened a new log
I20260927 15:06:18.217903 18977 ts_tablet_manager.cc:1397] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:06:18.218072 18978 ts_tablet_manager.cc:1397] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:06:18.219099 18979 tablet_bootstrap.cc:492] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: No bootstrap required, opened a new log
I20260927 15:06:18.219267 18979 ts_tablet_manager.cc:1397] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:06:18.220891 18977 raft_consensus.cc:348] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.221122 18977 raft_consensus.cc:374] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:18.221174 18977 raft_consensus.cc:729] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af71d798ea041e4805c9ede9e8894e4, State: Initialized, Role: FOLLOWER
I20260927 15:06:18.221604 18977 consensus_queue.cc:260] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.221840 18978 raft_consensus.cc:348] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.222149 18978 raft_consensus.cc:374] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:18.222214 18978 raft_consensus.cc:729] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c515f34b390240fea969f22552de6864, State: Initialized, Role: FOLLOWER
I20260927 15:06:18.222586 18977 ts_tablet_manager.cc:1428] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:06:18.222615 18978 consensus_queue.cc:260] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.222924 18826 heartbeater.cc:503] Master 127.0.98.62:39361 was elected leader, sending a full tablet report...
I20260927 15:06:18.222970 18979 raft_consensus.cc:348] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.223240 18979 raft_consensus.cc:374] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:18.223376 18979 raft_consensus.cc:729] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345a7cf658ef45949912752a335d8ba5, State: Initialized, Role: FOLLOWER
I20260927 15:06:18.223552 18978 ts_tablet_manager.cc:1428] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:06:18.223778 18979 consensus_queue.cc:260] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.223861 18692 heartbeater.cc:503] Master 127.0.98.62:39361 was elected leader, sending a full tablet report...
I20260927 15:06:18.224725 18979 ts_tablet_manager.cc:1428] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:18.225041 18960 heartbeater.cc:503] Master 127.0.98.62:39361 was elected leader, sending a full tablet report...
W20260927 15:06:18.401089 18961 tablet.cc:2369] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:18.402333 18985 raft_consensus.cc:482] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:18.402545 18985 raft_consensus.cc:504] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.403692 18985 leader_election.cc:288] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3af71d798ea041e4805c9ede9e8894e4 (127.0.98.2:46263), c515f34b390240fea969f22552de6864 (127.0.98.1:33281)
I20260927 15:06:18.407800 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.404312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45093 (local address 127.0.98.2:46263)
0927 15:06:18.404493 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.404497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.404514 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:06:18.404571 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.404576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.404636 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.404717 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:18.405021 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.405612 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.406765 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.407184 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.407264 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.407337 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.407597 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.407670 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:06:18.407840 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.404217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55341 (local address 127.0.98.1:33281)
0927 15:06:18.404358 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.404363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.404698 (+ 335us) server_negotiation.cc:407] Connection header received
0927 15:06:18.404806 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.404809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.404857 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.404945 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:18.405225 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.405752 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.406849 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.407215 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.407370 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.407433 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.407661 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.407730 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:06:18.407835 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.404124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46263 (local address 127.0.98.3:45093)
0927 15:06:18.404231 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.404304 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:18.404411 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:18.404746 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:18.404748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:18.404773 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:18.404903 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.404913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.405727 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.405730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.406644 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.406653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.407331 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.407337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.407444 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.407500 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 15:06:18.407567 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:06:18.407653 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":38}
I20260927 15:06:18.408036 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.404124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33281 (local address 127.0.98.3:55341)
0927 15:06:18.404570 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.404582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:18.404697 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:18.404958 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:18.404980 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:18.404999 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:18.405116 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.405122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.405924 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.405927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.406725 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.406737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.407393 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.407398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.407517 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.407534 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:06:18.407628 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:06:18.407689 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":149,"thread_start_us":140,"threads_started":1}
I20260927 15:06:18.408511 18780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270d45d627349c38c07e14cdb4634e0" candidate_uuid: "345a7cf658ef45949912752a335d8ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af71d798ea041e4805c9ede9e8894e4" is_pre_election: true
I20260927 15:06:18.408653 18646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270d45d627349c38c07e14cdb4634e0" candidate_uuid: "345a7cf658ef45949912752a335d8ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c515f34b390240fea969f22552de6864" is_pre_election: true
I20260927 15:06:18.408831 18780 raft_consensus.cc:2454] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345a7cf658ef45949912752a335d8ba5 in term 0.
I20260927 15:06:18.408880 18646 raft_consensus.cc:2454] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345a7cf658ef45949912752a335d8ba5 in term 0.
I20260927 15:06:18.409350 18849 leader_election.cc:302] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345a7cf658ef45949912752a335d8ba5, c515f34b390240fea969f22552de6864; no voters:
I20260927 15:06:18.409622 18985 raft_consensus.cc:2781] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:18.409703 18985 raft_consensus.cc:482] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:18.409745 18985 raft_consensus.cc:3037] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:18.410867 18985 raft_consensus.cc:504] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.411226 18985 leader_election.cc:288] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [CANDIDATE]: Term 1 election: Requested vote from peers 3af71d798ea041e4805c9ede9e8894e4 (127.0.98.2:46263), c515f34b390240fea969f22552de6864 (127.0.98.1:33281)
I20260927 15:06:18.411603 18780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270d45d627349c38c07e14cdb4634e0" candidate_uuid: "345a7cf658ef45949912752a335d8ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af71d798ea041e4805c9ede9e8894e4"
I20260927 15:06:18.411618 18646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0270d45d627349c38c07e14cdb4634e0" candidate_uuid: "345a7cf658ef45949912752a335d8ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c515f34b390240fea969f22552de6864"
I20260927 15:06:18.411780 18780 raft_consensus.cc:3037] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:18.411782 18646 raft_consensus.cc:3037] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:18.413237 18646 raft_consensus.cc:2454] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345a7cf658ef45949912752a335d8ba5 in term 1.
I20260927 15:06:18.413237 18780 raft_consensus.cc:2454] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345a7cf658ef45949912752a335d8ba5 in term 1.
I20260927 15:06:18.413597 18847 leader_election.cc:302] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 345a7cf658ef45949912752a335d8ba5, 3af71d798ea041e4805c9ede9e8894e4, c515f34b390240fea969f22552de6864; no voters:
I20260927 15:06:18.413820 18985 raft_consensus.cc:2781] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:18.414079 18985 raft_consensus.cc:686] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 1 LEADER]: Becoming Leader. State: Replica: 345a7cf658ef45949912752a335d8ba5, State: Running, Role: LEADER
I20260927 15:06:18.414438 18985 consensus_queue.cc:237] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } }
I20260927 15:06:18.417366 18500 catalog_manager.cc:5495] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345a7cf658ef45949912752a335d8ba5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "345a7cf658ef45949912752a335d8ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345a7cf658ef45949912752a335d8ba5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 } health_report { overall_health: UNKNOWN } } }
W20260927 15:06:18.427497 18827 tablet.cc:2369] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:18.436990 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.433289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.0.98.3:43513)
0927 15:06:18.433433 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.433438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.433452 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:06:18.433630 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.433632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.433688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.433870 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:18.434135 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.434647 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.435517 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.435743 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.435848 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.436151 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:18.436555 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:18.436715 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.436788 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.436827 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:06:18.443219 18780 raft_consensus.cc:1261] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 1 FOLLOWER]: Refusing update from remote peer 345a7cf658ef45949912752a335d8ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:18.443219 18646 raft_consensus.cc:1261] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 1 FOLLOWER]: Refusing update from remote peer 345a7cf658ef45949912752a335d8ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:18.443987 18987 consensus_queue.cc:1035] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af71d798ea041e4805c9ede9e8894e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:18.444173 18985 consensus_queue.cc:1035] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c515f34b390240fea969f22552de6864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:18.454224 18990 mvcc.cc:205] Tried to move back new op lower bound from 7333976385295503360 to 7333976385187909632. Current Snapshot: MvccSnapshot[applied={T|T < 7333976385295503360}]
I20260927 15:06:18.454815 18991 mvcc.cc:205] Tried to move back new op lower bound from 7333976385295503360 to 7333976385187909632. Current Snapshot: MvccSnapshot[applied={T|T < 7333976385295503360}]
I20260927 15:06:18.463091 18501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:18.463303 18501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:18.465806 18501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc9e374e13da4c2e8f74cf65c90924eb: Sending DeleteTablet for 3 replicas of tablet 0270d45d627349c38c07e14cdb4634e0
I20260927 15:06:18.466679 18760 tablet_service.cc:1514] Processing DeleteTablet for tablet 0270d45d627349c38c07e14cdb4634e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35398
I20260927 15:06:18.466693 18894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0270d45d627349c38c07e14cdb4634e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48026
I20260927 15:06:18.466742 18626 tablet_service.cc:1514] Processing DeleteTablet for tablet 0270d45d627349c38c07e14cdb4634e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39928
I20260927 15:06:18.467322 19002 tablet_replica.cc:330] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864: stopping tablet replica
I20260927 15:06:18.467628 19002 raft_consensus.cc:2229] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:18.467900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 18563
I20260927 15:06:18.467945 19002 raft_consensus.cc:2258] T 0270d45d627349c38c07e14cdb4634e0 P c515f34b390240fea969f22552de6864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:18.468544 19001 tablet_replica.cc:330] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: stopping tablet replica
I20260927 15:06:18.468873 19001 raft_consensus.cc:2229] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:18.469174 19001 raft_consensus.cc:2258] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:18.469569 19000 tablet_replica.cc:330] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: stopping tablet replica
I20260927 15:06:18.469897 19000 raft_consensus.cc:2229] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:18.470302 19000 raft_consensus.cc:2258] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:18.471179 19001 ts_tablet_manager.cc:1905] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:18.471693 19000 ts_tablet_manager.cc:1905] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:18.474190 19001 ts_tablet_manager.cc:1918] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:18.474372 19001 log.cc:1196] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/0270d45d627349c38c07e14cdb4634e0
I20260927 15:06:18.474965 19001 ts_tablet_manager.cc:1939] T 0270d45d627349c38c07e14cdb4634e0 P 3af71d798ea041e4805c9ede9e8894e4: Deleting consensus metadata
I20260927 15:06:18.475322 19000 ts_tablet_manager.cc:1918] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:18.475425 19000 log.cc:1196] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/0270d45d627349c38c07e14cdb4634e0
I20260927 15:06:18.475741 19000 ts_tablet_manager.cc:1939] T 0270d45d627349c38c07e14cdb4634e0 P 345a7cf658ef45949912752a335d8ba5: Deleting consensus metadata
I20260927 15:06:18.476583 18485 catalog_manager.cc:4841] TS 3af71d798ea041e4805c9ede9e8894e4 (127.0.98.2:46263): tablet 0270d45d627349c38c07e14cdb4634e0 (table test-table [id=dd2e7813eaf14549a38d5f61b0e78df9]) successfully deleted
I20260927 15:06:18.477047 18488 catalog_manager.cc:4841] TS 345a7cf658ef45949912752a335d8ba5 (127.0.98.3:43513): tablet 0270d45d627349c38c07e14cdb4634e0 (table test-table [id=dd2e7813eaf14549a38d5f61b0e78df9]) successfully deleted
W20260927 15:06:18.478379 18487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33281 (error 108)
I20260927 15:06:18.478622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 18697
I20260927 15:06:18.480044 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.479749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33281 (local address 127.0.0.1:39940)
0927 15:06:18.479878 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.479973 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:06:18.480273 18487 catalog_manager.cc:4584] TS c515f34b390240fea969f22552de6864 (127.0.98.1:33281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0270d45d627349c38c07e14cdb4634e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33281: connect: Connection refused (error 111)
I20260927 15:06:18.486070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 18831
I20260927 15:06:18.492554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 18469
2026-09-27T15:06:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2448 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T15:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:18Z Disabled control of system clock
I20260927 15:06:18.522329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46285
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:06:18.634416 19007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:18.634683 19007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:18.634719 19007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:18.638600 19007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:18.638654 19007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:18.638669 19007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:18.638685 19007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:18.638700 19007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:18.643409 19007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:18.644480 19007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:18.645820 19007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:18.654083 19013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:18.654049 19015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:18.654049 19012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:18.654366 19007 server_base.cc:1034] running on GCE node
I20260927 15:06:18.654989 19007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:18.656066 19007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:18.657261 19007 hybrid_clock.cc:648] HybridClock initialized: now 1790521578657230 us; error 51 us; skew 500 ppm
I20260927 15:06:18.659134 19007 webserver.cc:460] Webserver started at http://127.0.98.62:41407/ using document root <none> and password file <none>
I20260927 15:06:18.659710 19007 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:18.659801 19007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:18.660027 19007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:18.662148 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "980bf7b8bc1747c6b13bb15b41443dc1"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "8283f13ff6ddb5e62d135ad8be5d3d5e"
server_key_iv: "3a4f7118828dec88cc4bcf61cfc5d5e2"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.662727 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "980bf7b8bc1747c6b13bb15b41443dc1"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "8283f13ff6ddb5e62d135ad8be5d3d5e"
server_key_iv: "3a4f7118828dec88cc4bcf61cfc5d5e2"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.666612 19007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:06:18.669021 19021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:18.670327 19007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:18.670482 19007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "980bf7b8bc1747c6b13bb15b41443dc1"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "8283f13ff6ddb5e62d135ad8be5d3d5e"
server_key_iv: "3a4f7118828dec88cc4bcf61cfc5d5e2"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.670595 19007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:18.700894 19007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:18.701694 19007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:18.701930 19007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:18.708834 19007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46285
I20260927 15:06:18.708873 19073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46285 every 8 connection(s)
I20260927 15:06:18.709820 19007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:18.713016 19074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:18.718139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19007
I20260927 15:06:18.718267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:18.718575 392 external_mini_cluster.cc:1428] Setting key a8a9db15dcf79fcc073970f294771774
I20260927 15:06:18.718721 19074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1: Bootstrap starting.
I20260927 15:06:18.721235 19074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:18.722049 19074 log.cc:824] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:18.723874 19074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1: No bootstrap required, opened a new log
I20260927 15:06:18.727276 19074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260927 15:06:18.727509 19074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:18.727594 19074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980bf7b8bc1747c6b13bb15b41443dc1, State: Initialized, Role: FOLLOWER
I20260927 15:06:18.727720 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.719867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.0.98.62:46285)
0927 15:06:18.720519 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.720534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.720567 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:06:18.721181 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.721198 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.721645 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.722001 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:18.722460 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.723517 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.724465 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.724920 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.725099 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.726009 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:18.726014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:18.726025 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:18.726384 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:18.726446 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:18.726465 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:18.726569 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.726702 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.726945 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260927 15:06:18.728134 19074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260927 15:06:18.728312 19074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:18.728376 19074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:18.728467 19074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:18.729344 19074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260927 15:06:18.729713 19074 leader_election.cc:302] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980bf7b8bc1747c6b13bb15b41443dc1; no voters:
I20260927 15:06:18.730034 19074 leader_election.cc:288] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:18.730132 19079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:18.730378 19079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [term 1 LEADER]: Becoming Leader. State: Replica: 980bf7b8bc1747c6b13bb15b41443dc1, State: Running, Role: LEADER
I20260927 15:06:18.730693 19079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260927 15:06:18.731208 19074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:18.732415 19080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } } }
I20260927 15:06:18.732551 19080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:18.732813 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980bf7b8bc1747c6b13bb15b41443dc1. Latest consensus state: current_term: 1 leader_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980bf7b8bc1747c6b13bb15b41443dc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } } }
I20260927 15:06:18.732908 19088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:18.732936 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:18.736053 19088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:18.741284 19088 catalog_manager.cc:1347] Generated new cluster ID: b03b303f12214657833a78c41164aa2e
I20260927 15:06:18.741370 19088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:18.754999 19088 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:18.756299 19088 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:18.765360 19088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1: Generated new TSK 0
I20260927 15:06:18.766201 19088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:18.771121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:06:18.881083 19098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:18.881335 19098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:18.881382 19098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:18.885403 19098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:18.885452 19098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:18.885504 19098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:06:18.890272 19098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:18.891378 19098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:18.892776 19098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:18.902030 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:18.902037 19106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:18.902232 19098 server_base.cc:1034] running on GCE node
W20260927 15:06:18.902434 19103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:18.902855 19098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:18.903442 19098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:18.904599 19098 hybrid_clock.cc:648] HybridClock initialized: now 1790521578904575 us; error 35 us; skew 500 ppm
I20260927 15:06:18.906451 19098 webserver.cc:460] Webserver started at http://127.0.98.1:38489/ using document root <none> and password file <none>
I20260927 15:06:18.907053 19098 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:18.907136 19098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:18.907327 19098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:18.909327 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "e0e078c0c20248d3bb3e42593417a4a8"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "74522677b3788e8cee268d74b588b660"
server_key_iv: "a5d0b40b18242de79b8095f9f18123ff"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.909919 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e0e078c0c20248d3bb3e42593417a4a8"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "74522677b3788e8cee268d74b588b660"
server_key_iv: "a5d0b40b18242de79b8095f9f18123ff"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.913515 19098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:06:18.915990 19112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:18.917433 19098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:06:18.917532 19098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "e0e078c0c20248d3bb3e42593417a4a8"
format_stamp: "Formatted at 2026-09-27 15:06:18 on dist-test-slave-ml7t"
server_key: "74522677b3788e8cee268d74b588b660"
server_key_iv: "a5d0b40b18242de79b8095f9f18123ff"
server_key_version: "encryptionkey@0"
I20260927 15:06:18.917626 19098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:18.932915 19098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:18.933537 19098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:18.933686 19098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:18.934310 19098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:18.935227 19098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:18.935277 19098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:18.935319 19098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:18.935336 19098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:18.945011 19098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35091
I20260927 15:06:18.945039 19225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35091 every 8 connection(s)
I20260927 15:06:18.946117 19098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:18.948026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19098
I20260927 15:06:18.948158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:18.948460 392 external_mini_cluster.cc:1428] Setting key 5e780c5d9952a4a6c40ca75e9fa29c4a
I20260927 15:06:18.950663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:06:18.956943 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.948031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48271 (local address 127.0.98.62:46285)
0927 15:06:18.948205 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:18.948211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:18.949569 (+ 1358us) server_negotiation.cc:407] Connection header received
0927 15:06:18.951513 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:18.951516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:18.951556 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:18.951632 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:18.952695 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.953483 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.954722 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.955132 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.955276 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.956376 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:18.956382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:18.956387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:18.956404 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:18.956432 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:18.956446 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:18.956535 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:18.956661 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:06:18.956728 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:06:18.957715 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:18.948619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46285 (local address 127.0.98.1:48271)
0927 15:06:18.949428 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:18.949469 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:18.951272 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:18.951806 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:18.951818 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:18.951932 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:18.952541 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.952555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.953615 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.953618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.954602 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:18.954611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:18.955312 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:18.955315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:18.955452 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:18.956118 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:18.956119 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:18.956158 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:18.956217 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:18.956534 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:18.956539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:18.956542 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:06:18.956629 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:06:18.956834 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":156,"threads_started":1}
I20260927 15:06:18.958999 19226 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46285
I20260927 15:06:18.959228 19226 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:18.959761 19226 heartbeater.cc:511] Master 127.0.98.62:46285 requested a full tablet report, sending...
I20260927 15:06:18.961342 19036 ts_manager.cc:194] Registered new tserver with Master: e0e078c0c20248d3bb3e42593417a4a8 (127.0.98.1:35091)
I20260927 15:06:18.962920 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48271
W20260927 15:06:19.062520 19229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:19.062767 19229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:19.062803 19229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:19.066882 19229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:19.066939 19229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:19.066993 19229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:06:19.071628 19229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:19.072676 19229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:19.074108 19229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:19.083671 19234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.083669 19237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.083755 19235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:19.083958 19229 server_base.cc:1034] running on GCE node
I20260927 15:06:19.084425 19229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:19.085080 19229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:19.086258 19229 hybrid_clock.cc:648] HybridClock initialized: now 1790521579086233 us; error 39 us; skew 500 ppm
I20260927 15:06:19.088124 19229 webserver.cc:460] Webserver started at http://127.0.98.2:33241/ using document root <none> and password file <none>
I20260927 15:06:19.088697 19229 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:19.088786 19229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:19.089035 19229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:19.091099 19229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "da14fe5be5004392b976aea1e47facb5"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-ml7t"
server_key: "ff3293f5acc8f854dfe92be7d161da03"
server_key_iv: "2b653b991bdecfd4f505a27ddf4d1c54"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.091661 19229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da14fe5be5004392b976aea1e47facb5"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-ml7t"
server_key: "ff3293f5acc8f854dfe92be7d161da03"
server_key_iv: "2b653b991bdecfd4f505a27ddf4d1c54"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.095185 19229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:19.097533 19243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.098964 19229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:19.099087 19229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "da14fe5be5004392b976aea1e47facb5"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-ml7t"
server_key: "ff3293f5acc8f854dfe92be7d161da03"
server_key_iv: "2b653b991bdecfd4f505a27ddf4d1c54"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.099197 19229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:19.114876 19229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:19.115530 19229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:19.115720 19229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:19.116271 19229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:19.117174 19229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:19.117252 19229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.117352 19229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:19.117403 19229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.126991 19229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45763
I20260927 15:06:19.127013 19356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45763 every 8 connection(s)
I20260927 15:06:19.128044 19229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:19.137918 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.130338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41011 (local address 127.0.98.62:46285)
0927 15:06:19.130483 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.130488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.131563 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 15:06:19.133161 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.133165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.133244 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.133325 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:19.134320 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.134869 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.136008 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.136242 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.136352 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.137404 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:19.137407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:19.137410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:19.137426 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:19.137449 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:19.137458 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:19.137537 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.137689 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.137750 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:06:19.137952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19229
I20260927 15:06:19.138110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:19.138413 392 external_mini_cluster.cc:1428] Setting key d518b9df86e2d27ef5c301cdfb4bf029
I20260927 15:06:19.138876 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.130704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46285 (local address 127.0.98.2:41011)
0927 15:06:19.131408 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.131453 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:19.132911 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:19.133510 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:19.133517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:19.133606 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:19.134175 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.134188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.134998 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.135007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.135874 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.135882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.136395 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.136399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.136564 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.137135 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:19.137136 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:19.137174 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:19.137240 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:19.137538 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:19.137550 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:19.137558 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:06:19.137689 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 15:06:19.138012 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260927 15:06:19.140069 19357 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46285
I20260927 15:06:19.140311 19357 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:19.140441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--builtin_ntp_servers=127.0.98.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:06:19.140918 19357 heartbeater.cc:511] Master 127.0.98.62:46285 requested a full tablet report, sending...
I20260927 15:06:19.142022 19035 ts_manager.cc:194] Registered new tserver with Master: da14fe5be5004392b976aea1e47facb5 (127.0.98.2:45763)
I20260927 15:06:19.142588 19035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41011
W20260927 15:06:19.250244 19360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:19.250535 19360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:19.250608 19360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:19.254812 19360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:19.254894 19360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:19.254976 19360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:06:19.259753 19360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:19.260860 19360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:19.262287 19360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:19.271641 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.271672 19368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:19.271656 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:19.272035 19360 server_base.cc:1034] running on GCE node
I20260927 15:06:19.272432 19360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:19.273034 19360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:19.274197 19360 hybrid_clock.cc:648] HybridClock initialized: now 1790521579274161 us; error 46 us; skew 500 ppm
I20260927 15:06:19.275991 19360 webserver.cc:460] Webserver started at http://127.0.98.3:43403/ using document root <none> and password file <none>
I20260927 15:06:19.276543 19360 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:19.276613 19360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:19.276872 19360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:19.278883 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "536f989ae9d0413bacc4373442815100"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-ml7t"
server_key: "9fca17ec057bb94ea8ff1cde7bb3f2a7"
server_key_iv: "a5aed1761443eb0e19b007cc8c943cd9"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.279431 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "536f989ae9d0413bacc4373442815100"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-ml7t"
server_key: "9fca17ec057bb94ea8ff1cde7bb3f2a7"
server_key_iv: "a5aed1761443eb0e19b007cc8c943cd9"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.282920 19360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:19.285269 19374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.286609 19360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:19.286773 19360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "536f989ae9d0413bacc4373442815100"
format_stamp: "Formatted at 2026-09-27 15:06:19 on dist-test-slave-ml7t"
server_key: "9fca17ec057bb94ea8ff1cde7bb3f2a7"
server_key_iv: "a5aed1761443eb0e19b007cc8c943cd9"
server_key_version: "encryptionkey@0"
I20260927 15:06:19.286908 19360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:19.322194 19360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:19.322974 19360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:19.323185 19360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:19.323796 19360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:19.324710 19360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:19.324786 19360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.324860 19360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:19.324903 19360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:19.334322 19360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33299
I20260927 15:06:19.334347 19487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33299 every 8 connection(s)
I20260927 15:06:19.335256 19360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:19.337330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19360
I20260927 15:06:19.337469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:19.337739 392 external_mini_cluster.cc:1428] Setting key b5e03dc62f51936482d536f45199d88d
I20260927 15:06:19.344740 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.337185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43135 (local address 127.0.98.62:46285)
0927 15:06:19.337374 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.337379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.338675 (+ 1296us) server_negotiation.cc:407] Connection header received
0927 15:06:19.340094 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.340097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.340152 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.340244 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:19.341126 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.341672 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.342869 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.343099 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.343211 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.344238 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:19.344242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:19.344245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:19.344258 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:19.344279 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:19.344289 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:19.344378 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.344535 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.344595 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 15:06:19.345538 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.337745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46285 (local address 127.0.98.3:43135)
0927 15:06:19.338528 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.338580 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:19.339884 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:19.340417 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:19.340425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:19.340506 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:19.340989 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.341002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.341846 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.341850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.342744 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.342753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.343251 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.343256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.343415 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.343990 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:19.343991 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:19.344032 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:19.344094 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:19.344378 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:19.344389 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:19.344397 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:06:19.344525 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 15:06:19.344798 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
I20260927 15:06:19.346606 19488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46285
I20260927 15:06:19.346839 19488 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:19.347339 19488 heartbeater.cc:511] Master 127.0.98.62:46285 requested a full tablet report, sending...
I20260927 15:06:19.348322 19035 ts_manager.cc:194] Registered new tserver with Master: 536f989ae9d0413bacc4373442815100 (127.0.98.3:33299)
I20260927 15:06:19.348894 19035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43135
I20260927 15:06:19.351620 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:19.357270 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.353585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.0.98.62:46285)
0927 15:06:19.353732 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.353737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.353989 (+ 252us) server_negotiation.cc:407] Connection header received
0927 15:06:19.354072 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.354074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.354112 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.354177 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:19.354383 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.354896 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.355690 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.355907 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.355986 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.356768 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:19.356771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:19.356784 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:19.356796 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:19.356814 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:19.356821 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:19.356921 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.357040 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.357103 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:06:19.360500 19035 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:19.363576 19035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:19.378394 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.373159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.0.98.2:45763)
0927 15:06:19.373478 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.373489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.373512 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:06:19.373589 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.373594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.373808 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.374053 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:19.374197 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.375178 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.376779 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.377589 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.377701 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.377935 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.378116 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.378222 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260927 15:06:19.378388 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.372056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45763 (local address 127.0.0.1:33724)
0927 15:06:19.372757 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.372773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:19.372875 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:19.373968 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:19.373971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:19.373988 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:19.374114 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.374124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.375294 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.375300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.376630 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.376640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.377696 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.377699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.377898 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.377955 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 15:06:19.378081 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:06:19.378233 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":96,"thread_start_us":86,"threads_started":1}
I20260927 15:06:19.378393 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.371975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35091 (local address 127.0.0.1:47338)
0927 15:06:19.372332 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.372390 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:19.372509 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:19.373874 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:19.373880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:19.373904 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:19.374077 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.374085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.375605 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.375608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.376685 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.376694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.377714 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.377718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.377920 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.377966 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:06:19.378080 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:06:19.378222 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":36,"thread_start_us":70,"threads_started":1}
I20260927 15:06:19.379180 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.372301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33299 (local address 127.0.0.1:43836)
0927 15:06:19.373410 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.373424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:19.373513 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:19.374134 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:19.374136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:19.374151 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:19.374238 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.374246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.376800 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.376802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.377500 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.377507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.378839 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.378842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.378923 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.378943 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:06:19.379010 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:06:19.379072 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":84,"threads_started":1}
I20260927 15:06:19.379606 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.372787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.0.98.3:33299)
0927 15:06:19.373248 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.373257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.373604 (+ 347us) server_negotiation.cc:407] Connection header received
0927 15:06:19.373685 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.373689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.373875 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.374004 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:19.374365 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.375485 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.377635 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.378485 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.378595 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.378778 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.379343 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.379443 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":128,"threads_started":1}
I20260927 15:06:19.382018 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.372659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.0.98.1:35091)
0927 15:06:19.373290 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.373302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.373327 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:06:19.373404 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.373410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.373665 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.373900 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:19.374216 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.375485 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.376792 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.377589 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.378988 (+ 1399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.379195 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.379248 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.381893 (+ 2645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":139,"threads_started":1}
I20260927 15:06:19.381672 19291 tablet_service.cc:1467] Processing CreateTablet for tablet c8dbac6e05af40cbac59ca4457f2ac07 (DEFAULT_TABLE table=test-table [id=bb22bdf9193844dc92f3d90a87ff012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:19.382256 19422 tablet_service.cc:1467] Processing CreateTablet for tablet c8dbac6e05af40cbac59ca4457f2ac07 (DEFAULT_TABLE table=test-table [id=bb22bdf9193844dc92f3d90a87ff012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:19.382880 19291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8dbac6e05af40cbac59ca4457f2ac07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:19.383263 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8dbac6e05af40cbac59ca4457f2ac07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:19.384339 19160 tablet_service.cc:1467] Processing CreateTablet for tablet c8dbac6e05af40cbac59ca4457f2ac07 (DEFAULT_TABLE table=test-table [id=bb22bdf9193844dc92f3d90a87ff012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:19.385181 19160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8dbac6e05af40cbac59ca4457f2ac07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:19.390432 19505 tablet_bootstrap.cc:492] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Bootstrap starting.
I20260927 15:06:19.391338 19506 tablet_bootstrap.cc:492] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: Bootstrap starting.
I20260927 15:06:19.391980 19507 tablet_bootstrap.cc:492] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Bootstrap starting.
I20260927 15:06:19.393595 19507 tablet_bootstrap.cc:654] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:19.393600 19505 tablet_bootstrap.cc:654] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:19.393600 19506 tablet_bootstrap.cc:654] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:19.394352 19507 log.cc:824] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:19.394470 19506 log.cc:824] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:19.394470 19505 log.cc:824] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:19.396432 19505 tablet_bootstrap.cc:492] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: No bootstrap required, opened a new log
I20260927 15:06:19.396450 19507 tablet_bootstrap.cc:492] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: No bootstrap required, opened a new log
I20260927 15:06:19.396643 19507 ts_tablet_manager.cc:1397] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:06:19.396632 19505 ts_tablet_manager.cc:1397] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:06:19.397149 19506 tablet_bootstrap.cc:492] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: No bootstrap required, opened a new log
I20260927 15:06:19.397325 19506 ts_tablet_manager.cc:1397] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:06:19.399912 19507 raft_consensus.cc:348] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.400146 19507 raft_consensus.cc:374] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:19.400197 19507 raft_consensus.cc:729] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536f989ae9d0413bacc4373442815100, State: Initialized, Role: FOLLOWER
I20260927 15:06:19.400638 19507 consensus_queue.cc:260] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.400866 19505 raft_consensus.cc:348] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.400874 19506 raft_consensus.cc:348] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.401116 19506 raft_consensus.cc:374] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:19.401116 19505 raft_consensus.cc:374] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:19.401176 19505 raft_consensus.cc:729] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da14fe5be5004392b976aea1e47facb5, State: Initialized, Role: FOLLOWER
I20260927 15:06:19.401176 19506 raft_consensus.cc:729] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e078c0c20248d3bb3e42593417a4a8, State: Initialized, Role: FOLLOWER
I20260927 15:06:19.401413 19488 heartbeater.cc:503] Master 127.0.98.62:46285 was elected leader, sending a full tablet report...
I20260927 15:06:19.401671 19507 ts_tablet_manager.cc:1428] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:19.401625 19505 consensus_queue.cc:260] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.401625 19506 consensus_queue.cc:260] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.402567 19506 ts_tablet_manager.cc:1428] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:19.402642 19505 ts_tablet_manager.cc:1428] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:06:19.403026 19226 heartbeater.cc:503] Master 127.0.98.62:46285 was elected leader, sending a full tablet report...
I20260927 15:06:19.403422 19357 heartbeater.cc:503] Master 127.0.98.62:46285 was elected leader, sending a full tablet report...
W20260927 15:06:19.447399 19227 tablet.cc:2369] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:19.462531 19513 raft_consensus.cc:482] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:19.462852 19513 raft_consensus.cc:504] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.464151 19513 leader_election.cc:288] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da14fe5be5004392b976aea1e47facb5 (127.0.98.2:45763), 536f989ae9d0413bacc4373442815100 (127.0.98.3:33299)
I20260927 15:06:19.469539 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.464516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45005 (local address 127.0.98.2:45763)
0927 15:06:19.464656 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.464660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.465058 (+ 398us) server_negotiation.cc:407] Connection header received
0927 15:06:19.465236 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.465241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.465323 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.465409 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:19.466076 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.466858 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.468029 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.468613 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.468688 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.468749 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.469350 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.469423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:19.469558 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.464347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45763 (local address 127.0.98.1:45005)
0927 15:06:19.464966 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.464982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:19.465108 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:19.465708 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:19.465713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:19.465736 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:19.465932 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.465943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.466962 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.466968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.467885 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.467896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.468844 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.468848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.468946 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.468977 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:06:19.469313 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 15:06:19.469403 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260927 15:06:19.470238 19293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8dbac6e05af40cbac59ca4457f2ac07" candidate_uuid: "e0e078c0c20248d3bb3e42593417a4a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da14fe5be5004392b976aea1e47facb5" is_pre_election: true
I20260927 15:06:19.470599 19293 raft_consensus.cc:2454] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e078c0c20248d3bb3e42593417a4a8 in term 0.
I20260927 15:06:19.470820 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.464816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34435 (local address 127.0.98.3:33299)
0927 15:06:19.465021 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.465026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.465325 (+ 299us) server_negotiation.cc:407] Connection header received
0927 15:06:19.465426 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.465432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.465493 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.465574 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:19.465863 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.466649 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.468367 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.468942 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.469043 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.469094 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.470610 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.470701 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 15:06:19.470976 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.464444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33299 (local address 127.0.98.1:34435)
0927 15:06:19.465218 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.465231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:19.465330 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:19.465601 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:19.465603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:19.465617 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:19.465705 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.465709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.466768 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.466773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.468242 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:19.468255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.469061 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.469067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:19.470337 (+ 1270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.470440 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 15:06:19.470565 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:06:19.470664 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":68,"threads_started":1}
I20260927 15:06:19.471271 19114 leader_election.cc:302] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da14fe5be5004392b976aea1e47facb5, e0e078c0c20248d3bb3e42593417a4a8; no voters:
I20260927 15:06:19.471575 19513 raft_consensus.cc:2781] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:19.471704 19513 raft_consensus.cc:482] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:19.471761 19513 raft_consensus.cc:3037] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:19.471925 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8dbac6e05af40cbac59ca4457f2ac07" candidate_uuid: "e0e078c0c20248d3bb3e42593417a4a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536f989ae9d0413bacc4373442815100" is_pre_election: true
I20260927 15:06:19.472150 19442 raft_consensus.cc:2454] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e078c0c20248d3bb3e42593417a4a8 in term 0.
I20260927 15:06:19.473503 19513 raft_consensus.cc:504] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.474025 19513 leader_election.cc:288] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [CANDIDATE]: Term 1 election: Requested vote from peers da14fe5be5004392b976aea1e47facb5 (127.0.98.2:45763), 536f989ae9d0413bacc4373442815100 (127.0.98.3:33299)
I20260927 15:06:19.474397 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8dbac6e05af40cbac59ca4457f2ac07" candidate_uuid: "e0e078c0c20248d3bb3e42593417a4a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536f989ae9d0413bacc4373442815100"
I20260927 15:06:19.474418 19293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8dbac6e05af40cbac59ca4457f2ac07" candidate_uuid: "e0e078c0c20248d3bb3e42593417a4a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da14fe5be5004392b976aea1e47facb5"
I20260927 15:06:19.474534 19442 raft_consensus.cc:3037] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:19.474565 19293 raft_consensus.cc:3037] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:19.475723 19442 raft_consensus.cc:2454] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e078c0c20248d3bb3e42593417a4a8 in term 1.
I20260927 15:06:19.476081 19113 leader_election.cc:302] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536f989ae9d0413bacc4373442815100, e0e078c0c20248d3bb3e42593417a4a8; no voters:
I20260927 15:06:19.476150 19293 raft_consensus.cc:2454] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e078c0c20248d3bb3e42593417a4a8 in term 1.
I20260927 15:06:19.476270 19513 raft_consensus.cc:2781] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:19.476513 19513 raft_consensus.cc:686] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 1 LEADER]: Becoming Leader. State: Replica: e0e078c0c20248d3bb3e42593417a4a8, State: Running, Role: LEADER
I20260927 15:06:19.476851 19513 consensus_queue.cc:237] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } }
I20260927 15:06:19.480526 19038 catalog_manager.cc:5495] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e078c0c20248d3bb3e42593417a4a8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e0e078c0c20248d3bb3e42593417a4a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e078c0c20248d3bb3e42593417a4a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 } health_report { overall_health: UNKNOWN } } }
I20260927 15:06:19.504621 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.500124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.0.98.1:35091)
0927 15:06:19.500356 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:19.500364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:19.500385 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:06:19.500489 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:19.500492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:19.500556 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:19.500693 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:19.501006 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.501592 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.502688 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:19.503023 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:19.503197 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:19.503609 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:19.504088 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:19.504290 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:19.504396 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 15:06:19.504438 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 15:06:19.511183 19442 raft_consensus.cc:1261] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 1 FOLLOWER]: Refusing update from remote peer e0e078c0c20248d3bb3e42593417a4a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:19.511277 19293 raft_consensus.cc:1261] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 1 FOLLOWER]: Refusing update from remote peer e0e078c0c20248d3bb3e42593417a4a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:19.512127 19513 consensus_queue.cc:1035] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da14fe5be5004392b976aea1e47facb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:19.512385 19516 consensus_queue.cc:1035] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "536f989ae9d0413bacc4373442815100" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:19.520726 19519 mvcc.cc:205] Tried to move back new op lower bound from 7333976389670043648 to 7333976389539909632. Current Snapshot: MvccSnapshot[applied={T|T < 7333976389670043648}]
I20260927 15:06:19.530030 19520 mvcc.cc:205] Tried to move back new op lower bound from 7333976389670043648 to 7333976389539909632. Current Snapshot: MvccSnapshot[applied={T|T < 7333976389670043648}]
I20260927 15:06:19.535015 19035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:19.535233 19035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:19.538450 19035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 980bf7b8bc1747c6b13bb15b41443dc1: Sending DeleteTablet for 3 replicas of tablet c8dbac6e05af40cbac59ca4457f2ac07
I20260927 15:06:19.539367 19160 tablet_service.cc:1514] Processing DeleteTablet for tablet c8dbac6e05af40cbac59ca4457f2ac07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:19 UTC) from {username='slave'} at 127.0.0.1:47338
I20260927 15:06:19.540040 19528 tablet_replica.cc:330] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8: stopping tablet replica
I20260927 15:06:19.540017 19422 tablet_service.cc:1514] Processing DeleteTablet for tablet c8dbac6e05af40cbac59ca4457f2ac07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:19 UTC) from {username='slave'} at 127.0.0.1:43836
I20260927 15:06:19.540516 19528 raft_consensus.cc:2229] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:19.540763 19291 tablet_service.cc:1514] Processing DeleteTablet for tablet c8dbac6e05af40cbac59ca4457f2ac07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:19 UTC) from {username='slave'} at 127.0.0.1:33724
I20260927 15:06:19.541039 19528 raft_consensus.cc:2258] T c8dbac6e05af40cbac59ca4457f2ac07 P e0e078c0c20248d3bb3e42593417a4a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:19.541682 19529 tablet_replica.cc:330] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: stopping tablet replica
I20260927 15:06:19.542083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19098
I20260927 15:06:19.542099 19529 raft_consensus.cc:2229] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:19.545202 19530 tablet_replica.cc:330] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: stopping tablet replica
I20260927 15:06:19.545552 19530 raft_consensus.cc:2229] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:19.545945 19530 raft_consensus.cc:2258] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:19.546491 19529 raft_consensus.cc:2258] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:19.548337 19530 ts_tablet_manager.cc:1905] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:19.548355 19529 ts_tablet_manager.cc:1905] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:06:19.551353 19025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35091 (error 108)
I20260927 15:06:19.552289 19530 ts_tablet_manager.cc:1918] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:19.552356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19229
I20260927 15:06:19.552410 19530 log.cc:1196] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/c8dbac6e05af40cbac59ca4457f2ac07
I20260927 15:06:19.552783 19530 ts_tablet_manager.cc:1939] T c8dbac6e05af40cbac59ca4457f2ac07 P da14fe5be5004392b976aea1e47facb5: Deleting consensus metadata
I20260927 15:06:19.553448 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.552792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35091 (local address 127.0.0.1:47350)
0927 15:06:19.553236 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.553328 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
I20260927 15:06:19.553902 19529 ts_tablet_manager.cc:1918] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:19.554018 19529 log.cc:1196] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/c8dbac6e05af40cbac59ca4457f2ac07
W20260927 15:06:19.554144 19025 catalog_manager.cc:4584] TS e0e078c0c20248d3bb3e42593417a4a8 (127.0.98.1:35091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8dbac6e05af40cbac59ca4457f2ac07: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35091: connect: Connection refused (error 111)
I20260927 15:06:19.554387 19529 ts_tablet_manager.cc:1939] T c8dbac6e05af40cbac59ca4457f2ac07 P 536f989ae9d0413bacc4373442815100: Deleting consensus metadata
I20260927 15:06:19.555652 19022 catalog_manager.cc:4841] TS 536f989ae9d0413bacc4373442815100 (127.0.98.3:33299): tablet c8dbac6e05af40cbac59ca4457f2ac07 (table test-table [id=bb22bdf9193844dc92f3d90a87ff012f]) successfully deleted
I20260927 15:06:19.559291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19360
I20260927 15:06:19.560300 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:19.559489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45763 (local address 127.0.0.1:33738)
0927 15:06:19.560043 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:19.560117 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
I20260927 15:06:19.565485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19007
2026-09-27T15:06:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:19 dist-test-slave-ml7t krb5kdc[19534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:19 dist-test-slave-ml7t krb5kdc[19534](info): set up 2 sockets
Sep 27 15:06:19 dist-test-slave-ml7t krb5kdc[19534](info): commencing operation
krb5kdc: starting...
W20260927 15:06:20.609843 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:20 dist-test-slave-ml7t krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:20.732494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:20.840682 19547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:20.840920 19547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:20.840960 19547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:20.844800 19547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:20.844846 19547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:20.844866 19547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:20.844880 19547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:20.844897 19547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:20.849956 19547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:20.850844 19547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:20.851805 19547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:20.860081 19553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:20.860110 19555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:20.860110 19552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:20.860146 19547 server_base.cc:1034] running on GCE node
I20260927 15:06:20.860831 19547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:20.861879 19547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:20.863051 19547 hybrid_clock.cc:648] HybridClock initialized: now 1790521580863031 us; error 31 us; skew 500 ppm
Sep 27 15:06:20 dist-test-slave-ml7t krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:20.865690 19547 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:20.866573 19547 webserver.cc:460] Webserver started at http://127.0.98.62:46217/ using document root <none> and password file <none>
I20260927 15:06:20.867105 19547 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:20.867163 19547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:20.867436 19547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:20.869557 19547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "83ef06a9ab7d4f1ebc67ddbc26883611"
format_stamp: "Formatted at 2026-09-27 15:06:20 on dist-test-slave-ml7t"
server_key: "47eb2d40d3bdef863fae3d86fd180b45"
server_key_iv: "6c351c17b0d14e98401955e9e8181d9a"
server_key_version: "encryptionkey@0"
I20260927 15:06:20.870160 19547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "83ef06a9ab7d4f1ebc67ddbc26883611"
format_stamp: "Formatted at 2026-09-27 15:06:20 on dist-test-slave-ml7t"
server_key: "47eb2d40d3bdef863fae3d86fd180b45"
server_key_iv: "6c351c17b0d14e98401955e9e8181d9a"
server_key_version: "encryptionkey@0"
I20260927 15:06:20.873829 19547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:20.876313 19562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:20.878031 19547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:20.878156 19547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "83ef06a9ab7d4f1ebc67ddbc26883611"
format_stamp: "Formatted at 2026-09-27 15:06:20 on dist-test-slave-ml7t"
server_key: "47eb2d40d3bdef863fae3d86fd180b45"
server_key_iv: "6c351c17b0d14e98401955e9e8181d9a"
server_key_version: "encryptionkey@0"
I20260927 15:06:20.878325 19547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:20.890419 19547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:20.893217 19547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:20.893415 19547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:20.900499 19547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37797
I20260927 15:06:20.900521 19614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37797 every 8 connection(s)
I20260927 15:06:20.901427 19547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:20.903993 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:20.908528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19547
I20260927 15:06:20.908639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:20.908960 392 external_mini_cluster.cc:1428] Setting key 6dc1076af997c5ac158417acd732216f
I20260927 15:06:20.909413 19615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611: Bootstrap starting.
I20260927 15:06:20.911875 19615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:20.912832 19615 log.cc:824] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:20.914894 19615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611: No bootstrap required, opened a new log
Sep 27 15:06:20 dist-test-slave-ml7t krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:20.919191 19615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37797 } }
I20260927 15:06:20.919483 19615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:20.919569 19615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ef06a9ab7d4f1ebc67ddbc26883611, State: Initialized, Role: FOLLOWER
I20260927 15:06:20.920033 19615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37797 } }
I20260927 15:06:20.920150 19615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:20.920230 19615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:20.920347 19615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:20.921191 19615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37797 } }
I20260927 15:06:20.921519 19615 leader_election.cc:302] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83ef06a9ab7d4f1ebc67ddbc26883611; no voters:
I20260927 15:06:20.921859 19615 leader_election.cc:288] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:20.921942 19620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:20.922113 19620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [term 1 LEADER]: Becoming Leader. State: Replica: 83ef06a9ab7d4f1ebc67ddbc26883611, State: Running, Role: LEADER
I20260927 15:06:20.922403 19620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37797 } }
I20260927 15:06:20.923095 19615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:20.923529 19620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83ef06a9ab7d4f1ebc67ddbc26883611. Latest consensus state: current_term: 1 leader_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37797 } } }
I20260927 15:06:20.923638 19620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:20.924055 19621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ef06a9ab7d4f1ebc67ddbc26883611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37797 } } }
I20260927 15:06:20.924232 19621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:20.924250 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:20.910327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.0.98.62:37797)
0927 15:06:20.910785 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:20.910800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:20.910837 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:06:20.911477 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:20.911493 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:20.911811 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:20.912167 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:20.913082 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.914309 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.915473 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:20.915919 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:20.916034 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:20.918866 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:20.918871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:20.918883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:20.918908 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:20.920894 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:20.921375 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:20.921380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:20.921386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:20.921504 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:20.921808 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:20.921812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:20.921817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:20.922354 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:20.922579 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:20.922870 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:06:20.923177 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":148,"threads_started":1}
I20260927 15:06:20.925287 19624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:20.928802 19624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:20.934243 19624 catalog_manager.cc:1347] Generated new cluster ID: 053bba1c64004e4c86aa3de2790b81b0
I20260927 15:06:20.934408 19624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:20.944267 19624 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:20.945618 19624 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:20.954386 19624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611: Generated new TSK 0
I20260927 15:06:20.955135 19624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:06:20.986380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37797
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:21.102051 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:21.102327 19641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:21.102370 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:21.106277 19641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:21.106330 19641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:21.106386 19641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:06:21.111495 19641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:21.112612 19641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:21.113574 19641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:21.123044 19647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:21.123044 19646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:21.123265 19641 server_base.cc:1034] running on GCE node
W20260927 15:06:21.123044 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:21.123828 19641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:21.124606 19641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:21.125833 19641 hybrid_clock.cc:648] HybridClock initialized: now 1790521581125757 us; error 30 us; skew 500 ppm
Sep 27 15:06:21 dist-test-slave-ml7t krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:21.128926 19641 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:06:21.130038 19641 webserver.cc:460] Webserver started at http://127.0.98.1:33641/ using document root <none> and password file <none>
I20260927 15:06:21.130653 19641 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:21.130769 19641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:21.131034 19641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:21.133405 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "b8cac6d8ef6c48419f1de7d451a36689"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "af238efe7a069b1fbac407b8e1460c0b"
server_key_iv: "1e4d01dd1eaa51380a1c7ac486bb1056"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.134055 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b8cac6d8ef6c48419f1de7d451a36689"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "af238efe7a069b1fbac407b8e1460c0b"
server_key_iv: "1e4d01dd1eaa51380a1c7ac486bb1056"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.138058 19641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 15:06:21.140409 19656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.141728 19641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:21.141954 19641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "b8cac6d8ef6c48419f1de7d451a36689"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "af238efe7a069b1fbac407b8e1460c0b"
server_key_iv: "1e4d01dd1eaa51380a1c7ac486bb1056"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.142087 19641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:21.165252 19641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:21.168886 19641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:21.169099 19641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:21.169806 19641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:21.170964 19641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:21.171039 19641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.171106 19641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:21.171141 19641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.180737 19641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45203
I20260927 15:06:21.180754 19769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45203 every 8 connection(s)
I20260927 15:06:21.181748 19641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:21.183470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19641
I20260927 15:06:21.183619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:21.183938 392 external_mini_cluster.cc:1428] Setting key 8509a4d4502cb13590ee2d92cb6c2621
Sep 27 15:06:21 dist-test-slave-ml7t krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:06:21.196517 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.183620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56033 (local address 127.0.98.62:37797)
0927 15:06:21.183748 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.183753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.184782 (+ 1029us) server_negotiation.cc:407] Connection header received
0927 15:06:21.185878 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.185880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.185921 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.185993 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.187548 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.188449 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.189647 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.190064 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.190191 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.192632 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.192637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.192642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.192672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.194375 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.194891 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.194895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.194897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.194982 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.195320 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.195325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.195327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.195523 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.195635 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.196194 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.196343 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:06:21.197669 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.184098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37797 (local address 127.0.98.1:56033)
0927 15:06:21.184616 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.184673 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.185647 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.186165 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.186178 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.186706 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:21.187398 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.187416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.188567 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.188570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.189499 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.189512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.190203 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.190206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.190340 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.190983 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:21.190999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:21.192463 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:21.194477 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.194482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.194494 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.194779 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.195077 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.195081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.195083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.195222 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.195661 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:21.195667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:21.195946 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.196154 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.196558 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":160,"threads_started":1}
I20260927 15:06:21.199079 19770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37797
I20260927 15:06:21.199342 19770 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:21.199843 19770 heartbeater.cc:511] Master 127.0.98.62:37797 requested a full tablet report, sending...
I20260927 15:06:21.201653 19579 ts_manager.cc:194] Registered new tserver with Master: b8cac6d8ef6c48419f1de7d451a36689 (127.0.98.1:45203)
I20260927 15:06:21.203437 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:06:21.214082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37797
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:21.328301 19775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:21.328581 19775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:21.328627 19775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:21.332671 19775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:21.332746 19775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:21.332808 19775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:06:21.338167 19775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:21.339252 19775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:21.340307 19775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:21.350059 19780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:21.350072 19781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:21.350394 19775 server_base.cc:1034] running on GCE node
W20260927 15:06:21.350080 19783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:21.350886 19775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:21.351593 19775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:21.352779 19775 hybrid_clock.cc:648] HybridClock initialized: now 1790521581352753 us; error 48 us; skew 500 ppm
Sep 27 15:06:21 dist-test-slave-ml7t krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:21.355681 19775 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:06:21.356634 19775 webserver.cc:460] Webserver started at http://127.0.98.2:35995/ using document root <none> and password file <none>
I20260927 15:06:21.357203 19775 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:21.357336 19775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:21.357571 19775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:21.359716 19775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef0f48e1ebab4536ac0238892139985e"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "173486c4a9339d643e6758d18c1f66f1"
server_key_iv: "c71eb915f50fa012d15781046e319033"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.360291 19775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef0f48e1ebab4536ac0238892139985e"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "173486c4a9339d643e6758d18c1f66f1"
server_key_iv: "c71eb915f50fa012d15781046e319033"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.364038 19775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 15:06:21.366513 19790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.367681 19775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:06:21.367836 19775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "ef0f48e1ebab4536ac0238892139985e"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "173486c4a9339d643e6758d18c1f66f1"
server_key_iv: "c71eb915f50fa012d15781046e319033"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.367954 19775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:21.390591 19775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:21.393509 19775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:21.393734 19775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:21.394385 19775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:21.395246 19775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:21.395326 19775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.395404 19775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:21.395453 19775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.405082 19775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46653
I20260927 15:06:21.405117 19903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46653 every 8 connection(s)
I20260927 15:06:21.406106 19775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:21.410632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19775
I20260927 15:06:21.410805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:21.411103 392 external_mini_cluster.cc:1428] Setting key 3d1eacee8319b74e144d72fba6354cdb
Sep 27 15:06:21 dist-test-slave-ml7t krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:21.420251 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.407976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42089 (local address 127.0.98.62:37797)
0927 15:06:21.408119 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.408123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.408929 (+ 806us) server_negotiation.cc:407] Connection header received
0927 15:06:21.409859 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.409863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.409924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.410028 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.411341 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.412068 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.413469 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.413921 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.414063 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.416564 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.416575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.416578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.416602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.418168 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.418688 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.418692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.418694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.418775 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.419121 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.419126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.419128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.419313 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.419415 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.419952 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.420082 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:06:21.421238 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.408350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37797 (local address 127.0.98.2:42089)
0927 15:06:21.408780 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.408821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.409618 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.410203 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.410210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.410575 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:21.411138 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.411152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.412203 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.412207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.413317 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.413328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.414082 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.414085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.414240 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.414922 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:21.414938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:21.416386 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:21.418290 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.418296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.418313 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.418580 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.418874 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.418880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.418882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.419026 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.419432 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:21.419439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:21.419705 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.419913 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.420270 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260927 15:06:21.422658 19904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37797
I20260927 15:06:21.422931 19904 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:06:21.423834 19904 heartbeater.cc:511] Master 127.0.98.62:37797 requested a full tablet report, sending...
I20260927 15:06:21.425205 19579 ts_manager.cc:194] Registered new tserver with Master: ef0f48e1ebab4536ac0238892139985e (127.0.98.2:46653)
I20260927 15:06:21.426139 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:06:21.441425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37797
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:21.554346 19909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:21.554590 19909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:21.554633 19909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:21.558771 19909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:21.558825 19909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:21.558907 19909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:06:21.564117 19909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:21.565138 19909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:21.566210 19909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:21.575747 19915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:21.575826 19914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:21.575934 19909 server_base.cc:1034] running on GCE node
W20260927 15:06:21.575765 19917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:21.576589 19909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:21.577349 19909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:21.578527 19909 hybrid_clock.cc:648] HybridClock initialized: now 1790521581578498 us; error 41 us; skew 500 ppm
Sep 27 15:06:21 dist-test-slave-ml7t krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:21.581436 19909 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:06:21.582460 19909 webserver.cc:460] Webserver started at http://127.0.98.3:46047/ using document root <none> and password file <none>
I20260927 15:06:21.583035 19909 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:21.583096 19909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:21.583312 19909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:21.585350 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "136d20ff66b1493baaedacb4977cf51f"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "e1e6d643afec527c264eaf646a5e624a"
server_key_iv: "900f9a011b261d7e096bb485b0b0d312"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.585951 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "136d20ff66b1493baaedacb4977cf51f"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "e1e6d643afec527c264eaf646a5e624a"
server_key_iv: "900f9a011b261d7e096bb485b0b0d312"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.589838 19909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:06:21.592209 19924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.593444 19909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:21.593575 19909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "136d20ff66b1493baaedacb4977cf51f"
format_stamp: "Formatted at 2026-09-27 15:06:21 on dist-test-slave-ml7t"
server_key: "e1e6d643afec527c264eaf646a5e624a"
server_key_iv: "900f9a011b261d7e096bb485b0b0d312"
server_key_version: "encryptionkey@0"
I20260927 15:06:21.593683 19909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:21.606369 19909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:21.609176 19909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:21.609328 19909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:21.609903 19909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:21.610730 19909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:21.610785 19909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.610826 19909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:21.610841 19909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:21.620213 19909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34959
I20260927 15:06:21.620285 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34959 every 8 connection(s)
I20260927 15:06:21.621161 19909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:21.628178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 19909
I20260927 15:06:21.628376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:21.628677 392 external_mini_cluster.cc:1428] Setting key cbccfc6985c678560c64854e40744860
Sep 27 15:06:21 dist-test-slave-ml7t krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:21.635351 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.623174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45565 (local address 127.0.98.62:37797)
0927 15:06:21.623327 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.623331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.624393 (+ 1062us) server_negotiation.cc:407] Connection header received
0927 15:06:21.625393 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.625396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.625457 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.625562 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.626835 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.627375 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.628619 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.628903 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.629019 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.631865 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.631868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.631870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.631900 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.633313 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.633862 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.633865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.633866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.633965 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.634351 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.634354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.634355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.634541 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.634627 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.635106 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.635207 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:21.636170 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.623597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37797 (local address 127.0.98.3:45565)
0927 15:06:21.624140 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.624211 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.625129 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.625755 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.625764 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.626190 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:21.626680 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.626694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.627513 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.627517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.628486 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.628496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.629059 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.629063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.629280 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.630074 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:21.630095 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:21.631698 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:21.633422 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.633427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.633440 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.633689 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.634064 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:21.634066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:21.634068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:21.634196 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:21.634639 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:21.634644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:21.634897 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.635055 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.635329 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":113,"threads_started":1}
I20260927 15:06:21.637276 20038 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37797
I20260927 15:06:21.637492 20038 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:21.637998 20038 heartbeater.cc:511] Master 127.0.98.62:37797 requested a full tablet report, sending...
I20260927 15:06:21.638957 19579 ts_manager.cc:194] Registered new tserver with Master: 136d20ff66b1493baaedacb4977cf51f (127.0.98.3:34959)
I20260927 15:06:21.639496 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45565
I20260927 15:06:21.642828 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:21.652413 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.644924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.0.98.62:37797)
0927 15:06:21.645042 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.645046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.645163 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:06:21.645281 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.645284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.645320 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.645385 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:21.646145 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.646760 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.647710 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.647927 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.648006 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.649432 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:21.649434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:21.649436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:21.649457 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:21.650951 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.651359 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.651362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.651363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.651436 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:21.651667 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:21.651669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:21.651670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:21.651887 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:21.652000 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.652200 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.652298 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:06:21.655170 19579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:21.658300 19579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:21.672911 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.666696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45203 (local address 127.0.0.1:34314)
0927 15:06:21.667050 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.667101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.667226 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.668092 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.668102 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.668131 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:21.668286 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.668291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.669807 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.669810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.671207 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.671218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.672401 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.672404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.672495 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.672512 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.672569 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.672758 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":42,"thread_start_us":80,"threads_started":1}
I20260927 15:06:21.672981 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.667826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.0.98.2:46653)
0927 15:06:21.668275 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.668286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.668310 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:06:21.668388 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.668393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.668584 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.668734 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:21.668946 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.669942 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.671385 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.672142 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.672266 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.672412 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.672728 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.672824 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
I20260927 15:06:21.673398 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.667141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34959 (local address 127.0.0.1:58414)
0927 15:06:21.667970 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.667985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.668072 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.668917 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.668919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.668939 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:21.669028 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.669034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.670577 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.670580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.671679 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.671689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.672968 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.672975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.673105 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.673126 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.673200 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.673261 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260927 15:06:21.673491 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.667546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.0.98.3:34959)
0927 15:06:21.668298 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.668309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.668327 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:06:21.668389 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.668393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.668584 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.668734 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:21.669155 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.670437 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.671811 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.672630 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.672756 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.672901 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.673236 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.673346 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":136,"threads_started":1}
I20260927 15:06:21.673707 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.667425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46653 (local address 127.0.0.1:54554)
0927 15:06:21.668066 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.668079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.668144 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.668729 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.668733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.668750 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:21.668834 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.668839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.670090 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.670093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.671242 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.671260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.672285 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.672289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.672441 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.672486 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.672588 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.673583 (+ 995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":896,"thread_start_us":76,"threads_started":1}
I20260927 15:06:21.673852 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.666879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.98.1:45203)
0927 15:06:21.667538 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.667554 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.667579 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:06:21.667655 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.667661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.667911 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.668412 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:21.668536 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.669624 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.671332 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.672175 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.672688 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.672831 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.672876 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.672953 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":126,"threads_started":1}
I20260927 15:06:21.676543 19838 tablet_service.cc:1467] Processing CreateTablet for tablet 13612deb4861478daf3aa1bef9927c94 (DEFAULT_TABLE table=test-table [id=b3babf7c36b946e685e9411e805b9dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:21.676616 19972 tablet_service.cc:1467] Processing CreateTablet for tablet 13612deb4861478daf3aa1bef9927c94 (DEFAULT_TABLE table=test-table [id=b3babf7c36b946e685e9411e805b9dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:21.676616 19704 tablet_service.cc:1467] Processing CreateTablet for tablet 13612deb4861478daf3aa1bef9927c94 (DEFAULT_TABLE table=test-table [id=b3babf7c36b946e685e9411e805b9dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:21.677520 19838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13612deb4861478daf3aa1bef9927c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:21.677629 19704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13612deb4861478daf3aa1bef9927c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:21.677629 19972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13612deb4861478daf3aa1bef9927c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:21.683957 20055 tablet_bootstrap.cc:492] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: Bootstrap starting.
I20260927 15:06:21.685097 20056 tablet_bootstrap.cc:492] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: Bootstrap starting.
I20260927 15:06:21.686108 20057 tablet_bootstrap.cc:492] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Bootstrap starting.
I20260927 15:06:21.686417 20055 tablet_bootstrap.cc:654] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:21.687121 20056 tablet_bootstrap.cc:654] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:21.687515 20055 log.cc:824] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:21.688141 20057 tablet_bootstrap.cc:654] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:21.688297 20056 log.cc:824] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:21.689322 20057 log.cc:824] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:21.689581 20055 tablet_bootstrap.cc:492] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: No bootstrap required, opened a new log
I20260927 15:06:21.689807 20055 ts_tablet_manager.cc:1397] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:21.690361 20056 tablet_bootstrap.cc:492] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: No bootstrap required, opened a new log
I20260927 15:06:21.690547 20056 ts_tablet_manager.cc:1397] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 15:06:21.691085 20057 tablet_bootstrap.cc:492] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: No bootstrap required, opened a new log
I20260927 15:06:21.691283 20057 ts_tablet_manager.cc:1397] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:06:21.693727 20055 raft_consensus.cc:348] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.694043 20055 raft_consensus.cc:374] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:21.694137 20055 raft_consensus.cc:729] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0f48e1ebab4536ac0238892139985e, State: Initialized, Role: FOLLOWER
I20260927 15:06:21.693984 20057 raft_consensus.cc:348] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.694305 20057 raft_consensus.cc:374] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:21.694382 20057 raft_consensus.cc:729] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136d20ff66b1493baaedacb4977cf51f, State: Initialized, Role: FOLLOWER
I20260927 15:06:21.694232 20056 raft_consensus.cc:348] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.694448 20056 raft_consensus.cc:374] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:21.694504 20056 raft_consensus.cc:729] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8cac6d8ef6c48419f1de7d451a36689, State: Initialized, Role: FOLLOWER
I20260927 15:06:21.694695 20055 consensus_queue.cc:260] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.694828 20057 consensus_queue.cc:260] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.695101 20056 consensus_queue.cc:260] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.695791 20055 ts_tablet_manager.cc:1428] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:06:21.695793 20057 ts_tablet_manager.cc:1428] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 15:06:21.695998 20056 ts_tablet_manager.cc:1428] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:06:21.696156 19904 heartbeater.cc:503] Master 127.0.98.62:37797 was elected leader, sending a full tablet report...
I20260927 15:06:21.696158 20038 heartbeater.cc:503] Master 127.0.98.62:37797 was elected leader, sending a full tablet report...
I20260927 15:06:21.697101 19770 heartbeater.cc:503] Master 127.0.98.62:37797 was elected leader, sending a full tablet report...
W20260927 15:06:21.872205 20039 tablet.cc:2369] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:21.893354 20061 raft_consensus.cc:482] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:21.893604 20061 raft_consensus.cc:504] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.894913 20061 leader_election.cc:288] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8cac6d8ef6c48419f1de7d451a36689 (127.0.98.1:45203), 136d20ff66b1493baaedacb4977cf51f (127.0.98.3:34959)
I20260927 15:06:21.899791 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.895296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39213 (local address 127.0.98.1:45203)
0927 15:06:21.895447 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.895453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.895468 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:06:21.895544 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.895549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.895598 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.895677 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:21.895989 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.896741 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.898425 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.899047 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.899159 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.899223 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.899568 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.899655 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:06:21.899791 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.895110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45203 (local address 127.0.98.2:39213)
0927 15:06:21.895257 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.895276 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.895378 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.895716 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.895720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.895743 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:21.895882 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.895887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.896918 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.896922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.898294 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.898305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.899188 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.899246 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.899396 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.899433 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.899534 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.899633 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":7}
I20260927 15:06:21.899803 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.895493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49733 (local address 127.0.98.3:34959)
0927 15:06:21.895633 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.895640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.895725 (+ 85us) server_negotiation.cc:407] Connection header received
0927 15:06:21.895856 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.895858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.895897 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.895987 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:21.896272 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.896954 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.898425 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.899047 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.899159 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.899241 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.899582 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.899654 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:06:21.900162 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.895239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34959 (local address 127.0.98.2:49733)
0927 15:06:21.895612 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:21.895624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:21.895754 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:21.896031 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:21.896034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:21.896053 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:21.896164 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.896171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.897034 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.897037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.898294 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:21.898305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.899310 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.899314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:21.899435 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.899459 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:06:21.899945 (+ 486us) client_negotiation.cc:240] Negotiation successful
0927 15:06:21.900025 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"spinlock_wait_cycles":3456,"thread_start_us":90,"threads_started":1}
I20260927 15:06:21.900580 19724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13612deb4861478daf3aa1bef9927c94" candidate_uuid: "ef0f48e1ebab4536ac0238892139985e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8cac6d8ef6c48419f1de7d451a36689" is_pre_election: true
I20260927 15:06:21.900898 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13612deb4861478daf3aa1bef9927c94" candidate_uuid: "ef0f48e1ebab4536ac0238892139985e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136d20ff66b1493baaedacb4977cf51f" is_pre_election: true
I20260927 15:06:21.900933 19724 raft_consensus.cc:2454] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0f48e1ebab4536ac0238892139985e in term 0.
I20260927 15:06:21.901187 19992 raft_consensus.cc:2454] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0f48e1ebab4536ac0238892139985e in term 0.
I20260927 15:06:21.901522 19791 leader_election.cc:302] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cac6d8ef6c48419f1de7d451a36689, ef0f48e1ebab4536ac0238892139985e; no voters:
I20260927 15:06:21.901832 20061 raft_consensus.cc:2781] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:21.901939 20061 raft_consensus.cc:482] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:21.901990 20061 raft_consensus.cc:3037] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:21.903455 20061 raft_consensus.cc:504] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
I20260927 15:06:21.903868 20061 leader_election.cc:288] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [CANDIDATE]: Term 1 election: Requested vote from peers b8cac6d8ef6c48419f1de7d451a36689 (127.0.98.1:45203), 136d20ff66b1493baaedacb4977cf51f (127.0.98.3:34959)
I20260927 15:06:21.904336 19724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13612deb4861478daf3aa1bef9927c94" candidate_uuid: "ef0f48e1ebab4536ac0238892139985e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8cac6d8ef6c48419f1de7d451a36689"
I20260927 15:06:21.904477 19724 raft_consensus.cc:3037] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:21.905869 19724 raft_consensus.cc:2454] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0f48e1ebab4536ac0238892139985e in term 1.
I20260927 15:06:21.905893 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13612deb4861478daf3aa1bef9927c94" candidate_uuid: "ef0f48e1ebab4536ac0238892139985e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136d20ff66b1493baaedacb4977cf51f"
I20260927 15:06:21.906030 19992 raft_consensus.cc:3037] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:21.906188 19791 leader_election.cc:302] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cac6d8ef6c48419f1de7d451a36689, ef0f48e1ebab4536ac0238892139985e; no voters:
I20260927 15:06:21.906365 20061 raft_consensus.cc:2781] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:21.906574 20061 raft_consensus.cc:686] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 1 LEADER]: Becoming Leader. State: Replica: ef0f48e1ebab4536ac0238892139985e, State: Running, Role: LEADER
I20260927 15:06:21.907163 20061 consensus_queue.cc:237] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } }
W20260927 15:06:21.907291 19905 tablet.cc:2369] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:21.907325 19992 raft_consensus.cc:2454] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0f48e1ebab4536ac0238892139985e in term 1.
I20260927 15:06:21.910913 19578 catalog_manager.cc:5495] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0f48e1ebab4536ac0238892139985e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ef0f48e1ebab4536ac0238892139985e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0f48e1ebab4536ac0238892139985e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
W20260927 15:06:21.933166 19771 tablet.cc:2369] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:21.965339 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:21.961931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.0.98.2:46653)
0927 15:06:21.962077 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:21.962081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:21.962093 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:06:21.962169 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:21.962171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:21.962224 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:21.962361 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:21.962634 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.963139 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.964028 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:21.964242 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:21.964328 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:21.964579 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:21.965016 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:21.965105 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:21.965164 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 15:06:21.965212 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:06:21.971935 19992 raft_consensus.cc:1261] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 1 FOLLOWER]: Refusing update from remote peer ef0f48e1ebab4536ac0238892139985e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:21.971935 19724 raft_consensus.cc:1261] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 1 FOLLOWER]: Refusing update from remote peer ef0f48e1ebab4536ac0238892139985e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:21.972739 20061 consensus_queue.cc:1035] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [LEADER]: Connected to new peer: Peer: permanent_uuid: "136d20ff66b1493baaedacb4977cf51f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:21.972976 20065 consensus_queue.cc:1035] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8cac6d8ef6c48419f1de7d451a36689" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:21.980590 20068 mvcc.cc:205] Tried to move back new op lower bound from 7333976399748169728 to 7333976399496032256. Current Snapshot: MvccSnapshot[applied={T|T < 7333976399748169728}]
I20260927 15:06:21.990054 19578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:21.990221 19578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:21.992594 19578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83ef06a9ab7d4f1ebc67ddbc26883611: Sending DeleteTablet for 3 replicas of tablet 13612deb4861478daf3aa1bef9927c94
I20260927 15:06:21.993391 19972 tablet_service.cc:1514] Processing DeleteTablet for tablet 13612deb4861478daf3aa1bef9927c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58414
I20260927 15:06:21.993500 19704 tablet_service.cc:1514] Processing DeleteTablet for tablet 13612deb4861478daf3aa1bef9927c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34314
I20260927 15:06:21.993626 19838 tablet_service.cc:1514] Processing DeleteTablet for tablet 13612deb4861478daf3aa1bef9927c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54554
I20260927 15:06:21.994077 20078 tablet_replica.cc:330] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689: stopping tablet replica
I20260927 15:06:21.994182 20077 tablet_replica.cc:330] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: stopping tablet replica
I20260927 15:06:21.994381 20078 raft_consensus.cc:2229] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:21.994459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19641
I20260927 15:06:21.994668 20078 raft_consensus.cc:2258] T 13612deb4861478daf3aa1bef9927c94 P b8cac6d8ef6c48419f1de7d451a36689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:21.994493 20077 raft_consensus.cc:2229] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:21.995785 20077 raft_consensus.cc:2258] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:21.997673 20077 ts_tablet_manager.cc:1905] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:22.000862 20077 ts_tablet_manager.cc:1918] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:22.000975 20077 log.cc:1196] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/13612deb4861478daf3aa1bef9927c94
I20260927 15:06:22.001432 20079 tablet_replica.cc:330] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e: stopping tablet replica
W20260927 15:06:22.001499 19563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45203 (error 108)
I20260927 15:06:22.001716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19775
I20260927 15:06:22.001971 20079 raft_consensus.cc:2229] T 13612deb4861478daf3aa1bef9927c94 P ef0f48e1ebab4536ac0238892139985e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:22.002090 20077 ts_tablet_manager.cc:1939] T 13612deb4861478daf3aa1bef9927c94 P 136d20ff66b1493baaedacb4977cf51f: Deleting consensus metadata
I20260927 15:06:22.003223 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.002882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45203 (local address 127.0.0.1:34324)
0927 15:06:22.003032 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.003112 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:06:22.003386 19563 catalog_manager.cc:4584] TS b8cac6d8ef6c48419f1de7d451a36689 (127.0.98.1:45203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13612deb4861478daf3aa1bef9927c94: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45203: connect: Connection refused (error 111)
I20260927 15:06:22.004249 19564 catalog_manager.cc:4841] TS 136d20ff66b1493baaedacb4977cf51f (127.0.98.3:34959): tablet 13612deb4861478daf3aa1bef9927c94 (table test-table [id=b3babf7c36b946e685e9411e805b9dc1]) successfully deleted
I20260927 15:06:22.008388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19909
I20260927 15:06:22.009384 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.008643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46653 (local address 127.0.0.1:54560)
0927 15:06:22.009213 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.009280 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468}
I20260927 15:06:22.014325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 19547
2026-09-27T15:06:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2449 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T15:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:22Z Disabled control of system clock
I20260927 15:06:22.044683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41417
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:06:22.152763 20084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:22.153071 20084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:22.153110 20084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:22.157106 20084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:22.157169 20084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:22.157186 20084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:22.157202 20084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:22.157217 20084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:22.161928 20084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:22.162935 20084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:22.164225 20084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:22.172498 20090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.172542 20092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.172551 20089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:22.172655 20084 server_base.cc:1034] running on GCE node
I20260927 15:06:22.173269 20084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:22.174748 20084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:22.175951 20084 hybrid_clock.cc:648] HybridClock initialized: now 1790521582175921 us; error 45 us; skew 500 ppm
I20260927 15:06:22.177969 20084 webserver.cc:460] Webserver started at http://127.0.98.62:36213/ using document root <none> and password file <none>
I20260927 15:06:22.178558 20084 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:22.178650 20084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:22.178893 20084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:22.180976 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "0c677dc9f26840338996bf11accb20cb"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "048fb8cfa4568c7674710b16c825cb0f"
server_key_iv: "a2cee942fae9afb8d90f6ef37c39ba86"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.181628 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c677dc9f26840338996bf11accb20cb"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "048fb8cfa4568c7674710b16c825cb0f"
server_key_iv: "a2cee942fae9afb8d90f6ef37c39ba86"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.185590 20084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:06:22.188098 20098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.189517 20084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:06:22.189662 20084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "0c677dc9f26840338996bf11accb20cb"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "048fb8cfa4568c7674710b16c825cb0f"
server_key_iv: "a2cee942fae9afb8d90f6ef37c39ba86"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.189859 20084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:22.213114 20084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:22.214079 20084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:22.214316 20084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:22.221697 20084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41417
I20260927 15:06:22.221704 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41417 every 8 connection(s)
I20260927 15:06:22.222747 20084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:22.225919 20151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:22.230372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20084
I20260927 15:06:22.230508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:22.230814 392 external_mini_cluster.cc:1428] Setting key 2ea592e58e7ca65c5e5b213ce20fe125
I20260927 15:06:22.232090 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb: Bootstrap starting.
I20260927 15:06:22.234465 20151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:22.235214 20151 log.cc:824] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:22.237041 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb: No bootstrap required, opened a new log
I20260927 15:06:22.240698 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.232306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.0.98.62:41417)
0927 15:06:22.232956 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.232971 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.233006 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:06:22.233665 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.233683 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.234164 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.234515 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:22.234964 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.236035 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.236915 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.237485 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.237657 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.238579 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:22.238586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:22.238605 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:22.238981 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:22.239035 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:22.239054 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:22.239163 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.239309 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.239591 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260927 15:06:22.240672 20151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c677dc9f26840338996bf11accb20cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41417 } }
I20260927 15:06:22.241016 20151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:22.241067 20151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c677dc9f26840338996bf11accb20cb, State: Initialized, Role: FOLLOWER
I20260927 15:06:22.241511 20151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c677dc9f26840338996bf11accb20cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41417 } }
I20260927 15:06:22.241645 20151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:22.241701 20151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:22.241822 20151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:22.242679 20151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c677dc9f26840338996bf11accb20cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41417 } }
I20260927 15:06:22.243048 20151 leader_election.cc:302] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c677dc9f26840338996bf11accb20cb; no voters:
I20260927 15:06:22.243361 20151 leader_election.cc:288] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:22.243456 20156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:22.243678 20156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [term 1 LEADER]: Becoming Leader. State: Replica: 0c677dc9f26840338996bf11accb20cb, State: Running, Role: LEADER
I20260927 15:06:22.243989 20156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c677dc9f26840338996bf11accb20cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41417 } }
I20260927 15:06:22.244489 20151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:22.245617 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c677dc9f26840338996bf11accb20cb. Latest consensus state: current_term: 1 leader_uuid: "0c677dc9f26840338996bf11accb20cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c677dc9f26840338996bf11accb20cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41417 } } }
I20260927 15:06:22.245653 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c677dc9f26840338996bf11accb20cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c677dc9f26840338996bf11accb20cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41417 } } }
I20260927 15:06:22.245787 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:22.245787 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:22.246197 20165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:22.249289 20165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:22.254837 20165 catalog_manager.cc:1347] Generated new cluster ID: 988ae436c8d847f193e039ebcc421326
I20260927 15:06:22.254932 20165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:22.284371 20165 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:22.285357 20165 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:22.291741 20165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb: Generated new TSK 0
I20260927 15:06:22.292616 20165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:22.306567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41417
--builtin_ntp_servers=127.0.98.20:33497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:06:22.422000 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:22.422331 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:22.422377 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:22.426246 20175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:22.426290 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:22.426342 20175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:06:22.430858 20175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:22.431857 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:22.433262 20175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:22.442984 20183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.442992 20181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.442996 20180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:22.443362 20175 server_base.cc:1034] running on GCE node
I20260927 15:06:22.443727 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:22.444367 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:22.445549 20175 hybrid_clock.cc:648] HybridClock initialized: now 1790521582445529 us; error 33 us; skew 500 ppm
I20260927 15:06:22.453372 20175 webserver.cc:460] Webserver started at http://127.0.98.1:36293/ using document root <none> and password file <none>
I20260927 15:06:22.454079 20175 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:22.454177 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:22.454480 20175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:22.457653 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data/instance:
uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "bd918ede30fe343ae9102df6522b14d2"
server_key_iv: "5339a2c200c288c5644397c9a0d17f77"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.458370 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "bd918ede30fe343ae9102df6522b14d2"
server_key_iv: "5339a2c200c288c5644397c9a0d17f77"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.462306 20175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:06:22.464939 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.466305 20175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:22.466442 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/wal
uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "bd918ede30fe343ae9102df6522b14d2"
server_key_iv: "5339a2c200c288c5644397c9a0d17f77"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.466560 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:22.486083 20175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:22.486809 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:22.487011 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:22.487632 20175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:22.488596 20175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:22.488667 20175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.488744 20175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:22.488790 20175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.498775 20175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33397
I20260927 15:06:22.498798 20302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33397 every 8 connection(s)
I20260927 15:06:22.499720 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:06:22.503692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20175
I20260927 15:06:22.503881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:06:22.504261 392 external_mini_cluster.cc:1428] Setting key 97bba4f41ad41e10c33a07dc78013ef8
I20260927 15:06:22.506817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41417
--builtin_ntp_servers=127.0.98.20:33497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:06:22.511914 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.501639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51025 (local address 127.0.98.62:41417)
0927 15:06:22.501826 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.501831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.502932 (+ 1101us) server_negotiation.cc:407] Connection header received
0927 15:06:22.504713 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.504717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.504810 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.504899 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:22.506475 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.507336 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.509245 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.509694 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.509824 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.511341 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:22.511345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:22.511347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:22.511362 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:22.511384 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:22.511395 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:22.511464 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.511688 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.511763 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:06:22.513049 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.502130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41417 (local address 127.0.98.1:51025)
0927 15:06:22.502777 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.502819 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:22.504471 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:22.505195 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:22.505204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:22.505368 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:22.506312 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.506334 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.507498 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.507505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.509105 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.509117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.509915 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.509921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.510196 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.511012 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:22.511014 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:22.511077 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:22.511164 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:22.511502 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:22.511512 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:22.511520 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:06:22.511645 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:06:22.511967 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260927 15:06:22.514460 20303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41417
I20260927 15:06:22.514729 20303 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:22.515369 20303 heartbeater.cc:511] Master 127.0.98.62:41417 requested a full tablet report, sending...
I20260927 15:06:22.517108 20115 ts_manager.cc:194] Registered new tserver with Master: ceb26eeb5f2440ba9b2ef3ab8fdab296 (127.0.98.1:33397)
I20260927 15:06:22.518615 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51025
W20260927 15:06:22.633244 20306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:22.633486 20306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:22.633522 20306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:22.637614 20306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:22.637672 20306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:22.637733 20306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:06:22.642491 20306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:22.643502 20306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:22.645046 20306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:22.655020 20312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.655026 20314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.655020 20311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:22.655081 20306 server_base.cc:1034] running on GCE node
I20260927 15:06:22.655689 20306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:22.656318 20306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:22.657506 20306 hybrid_clock.cc:648] HybridClock initialized: now 1790521582657472 us; error 45 us; skew 500 ppm
I20260927 15:06:22.659368 20306 webserver.cc:460] Webserver started at http://127.0.98.2:37239/ using document root <none> and password file <none>
I20260927 15:06:22.659946 20306 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:22.660037 20306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:22.660264 20306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:22.662361 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data/instance:
uuid: "3a03cdb5475445c5ad5ab31470c5fff9"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "9587243b4395ce9d78f96483948b4f8d"
server_key_iv: "b011c5e987a4f6e5333cb4449b9fb2f2"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.662931 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3a03cdb5475445c5ad5ab31470c5fff9"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "9587243b4395ce9d78f96483948b4f8d"
server_key_iv: "b011c5e987a4f6e5333cb4449b9fb2f2"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.666746 20306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:06:22.669133 20320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.670625 20306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:06:22.670713 20306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal
uuid: "3a03cdb5475445c5ad5ab31470c5fff9"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "9587243b4395ce9d78f96483948b4f8d"
server_key_iv: "b011c5e987a4f6e5333cb4449b9fb2f2"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.670796 20306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:22.701929 20306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:22.702597 20306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:22.702742 20306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:22.703271 20306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:22.704139 20306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:22.704186 20306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.704236 20306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:22.704252 20306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.713057 20306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35781
I20260927 15:06:22.713128 20433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35781 every 8 connection(s)
I20260927 15:06:22.714011 20306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:06:22.714881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20306
I20260927 15:06:22.715021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:06:22.715309 392 external_mini_cluster.cc:1428] Setting key bfad0e1169bfe4b752d34ea9bea165a7
I20260927 15:06:22.717670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41417
--builtin_ntp_servers=127.0.98.20:33497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:06:22.725241 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.716131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53823 (local address 127.0.98.62:41417)
0927 15:06:22.716260 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.716266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.717737 (+ 1471us) server_negotiation.cc:407] Connection header received
0927 15:06:22.719776 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.719781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.719842 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.719944 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:22.720878 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.721812 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.722976 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.723409 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.723544 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.724638 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:22.724645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:22.724651 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:22.724670 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:22.724696 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:22.724709 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:22.724805 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.724958 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.725032 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:06:22.726085 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.716657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41417 (local address 127.0.98.2:53823)
0927 15:06:22.717553 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.717611 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:22.719551 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:22.720099 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:22.720106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:22.720238 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:22.720723 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.720736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.721946 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.721949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.722846 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.722855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.723575 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.723578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.723725 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.724375 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:22.724376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:22.724415 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:22.724477 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:22.724814 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:22.724821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:22.724825 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:06:22.724930 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 15:06:22.725139 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":136,"threads_started":1}
I20260927 15:06:22.727332 20434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41417
I20260927 15:06:22.727542 20434 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:22.728051 20434 heartbeater.cc:511] Master 127.0.98.62:41417 requested a full tablet report, sending...
I20260927 15:06:22.729101 20115 ts_manager.cc:194] Registered new tserver with Master: 3a03cdb5475445c5ad5ab31470c5fff9 (127.0.98.2:35781)
I20260927 15:06:22.729660 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53823
W20260927 15:06:22.828264 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:22.828523 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:22.828562 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:22.832556 20437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:22.832609 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:22.832664 20437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:06:22.837330 20437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:22.838406 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:22.839793 20437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:22.849486 20445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.849524 20442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:22.849495 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:22.849530 20437 server_base.cc:1034] running on GCE node
I20260927 15:06:22.850119 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:22.850764 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:22.851963 20437 hybrid_clock.cc:648] HybridClock initialized: now 1790521582851943 us; error 34 us; skew 500 ppm
I20260927 15:06:22.853749 20437 webserver.cc:460] Webserver started at http://127.0.98.3:41297/ using document root <none> and password file <none>
I20260927 15:06:22.854393 20437 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:22.854480 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:22.854704 20437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:22.856649 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data/instance:
uuid: "ab944321633b489ab2f81ab1e7870ee4"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "adee5dc097df740d6f334e2a0eb548ab"
server_key_iv: "13ecc98d9c0cb9859f57a03666d1191b"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.857178 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ab944321633b489ab2f81ab1e7870ee4"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "adee5dc097df740d6f334e2a0eb548ab"
server_key_iv: "13ecc98d9c0cb9859f57a03666d1191b"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.860730 20437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 15:06:22.863168 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.864419 20437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:22.864563 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal
uuid: "ab944321633b489ab2f81ab1e7870ee4"
format_stamp: "Formatted at 2026-09-27 15:06:22 on dist-test-slave-ml7t"
server_key: "adee5dc097df740d6f334e2a0eb548ab"
server_key_iv: "13ecc98d9c0cb9859f57a03666d1191b"
server_key_version: "encryptionkey@0"
I20260927 15:06:22.864676 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:22.892725 20437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:22.893568 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:22.893823 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:22.894510 20437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:06:22.895481 20437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:06:22.895552 20437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.895618 20437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:06:22.895668 20437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:22.905083 20437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36375
I20260927 15:06:22.905112 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36375 every 8 connection(s)
I20260927 15:06:22.906041 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:06:22.914445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20437
I20260927 15:06:22.914579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:06:22.914870 392 external_mini_cluster.cc:1428] Setting key 87c477eabdf55e2745196400249f6281
I20260927 15:06:22.915679 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.907903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57735 (local address 127.0.98.62:41417)
0927 15:06:22.908055 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.908060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.909192 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 15:06:22.910651 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.910654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.910703 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.910801 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:22.911737 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.912299 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.913399 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.913630 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.913753 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.915141 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:22.915145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:22.915147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:22.915159 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:22.915178 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:22.915187 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:22.915278 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.915469 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.915536 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:06:22.916532 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.908403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41417 (local address 127.0.98.3:57735)
0927 15:06:22.909043 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.909082 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:22.910428 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:22.910973 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:22.910981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:22.911066 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:22.911585 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.911598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.912414 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.912416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.913260 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.913279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.913869 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.913874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.914027 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.914820 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:06:22.914821 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:22.914863 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:06:22.914960 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:22.915309 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:22.915318 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:22.915324 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:06:22.915454 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:06:22.915755 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260927 15:06:22.917582 20565 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41417
I20260927 15:06:22.917866 20565 heartbeater.cc:464] Registering TS with master...
I20260927 15:06:22.918414 20565 heartbeater.cc:511] Master 127.0.98.62:41417 requested a full tablet report, sending...
I20260927 15:06:22.919476 20115 ts_manager.cc:194] Registered new tserver with Master: ab944321633b489ab2f81ab1e7870ee4 (127.0.98.3:36375)
I20260927 15:06:22.920024 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57735
I20260927 15:06:22.928578 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:06:22.934131 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.930604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.0.98.62:41417)
0927 15:06:22.930719 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.930728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.930858 (+ 130us) server_negotiation.cc:407] Connection header received
0927 15:06:22.930925 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.930927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.930967 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.931035 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:22.931264 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.931761 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.932654 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.932884 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.932971 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.933729 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:22.933747 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:22.933749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:22.933760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:22.933841 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:22.933851 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:22.933912 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.933985 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.934031 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:06:22.937072 20115 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:06:22.940150 20115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:06:22.954766 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.948775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36375 (local address 127.0.0.1:51328)
0927 15:06:22.949801 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.949813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:22.949903 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:22.950654 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:22.950658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:22.950676 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:22.950767 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.950772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.951873 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.951878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.952920 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.952929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.954172 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.954177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.954303 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.954348 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:06:22.954471 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:06:22.954605 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":95,"threads_started":1}
I20260927 15:06:22.954788 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.949032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.98.3:36375)
0927 15:06:22.949648 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.949666 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.950024 (+ 358us) server_negotiation.cc:407] Connection header received
0927 15:06:22.950108 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.950114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.950350 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.950500 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:22.950882 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.951765 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.953050 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.954044 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.954150 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.954347 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.954524 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.954618 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":177,"threads_started":1}
I20260927 15:06:22.955498 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.949002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33397 (local address 127.0.0.1:42172)
0927 15:06:22.950145 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.950161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:22.950308 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:22.951025 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:22.951027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:22.951044 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:22.951117 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.951122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.952657 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.952661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.953660 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.953670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.955119 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.955125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.955232 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.955258 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:06:22.955336 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:06:22.955404 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":100,"threads_started":1}
I20260927 15:06:22.955889 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.948761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35781 (local address 127.0.0.1:55434)
0927 15:06:22.949614 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:22.949663 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:22.949822 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:22.950754 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:22.950757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:22.950777 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:22.950891 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.950900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.953076 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.953079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.954192 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:22.954201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.955510 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.955514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:22.955597 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.955617 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:06:22.955686 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:06:22.955755 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":102,"threads_started":1}
I20260927 15:06:22.956089 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.949590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.0.98.1:33397)
0927 15:06:22.950590 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.950604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.950626 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:06:22.950702 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.950710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.950880 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.951039 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:22.951266 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.952464 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.953824 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.954544 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.954647 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.954845 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.955774 (+ 929us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.955877 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":117,"threads_started":1}
I20260927 15:06:22.956300 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:22.949494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.0.98.2:35781)
0927 15:06:22.950076 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:22.950093 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:22.950119 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:06:22.950208 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:22.950214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:22.950420 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:22.950577 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:22.951238 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.952468 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.954336 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:22.955142 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:22.955266 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:22.955443 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:22.956045 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 15:06:22.956143 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":103,"threads_started":1}
I20260927 15:06:22.958084 20499 tablet_service.cc:1467] Processing CreateTablet for tablet cd4569304ba946d0b6a348f665669d8a (DEFAULT_TABLE table=test-table [id=59287bdf11bb4470a474c2c43fe3ea47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:22.958653 20368 tablet_service.cc:1467] Processing CreateTablet for tablet cd4569304ba946d0b6a348f665669d8a (DEFAULT_TABLE table=test-table [id=59287bdf11bb4470a474c2c43fe3ea47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:22.959183 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4569304ba946d0b6a348f665669d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:22.958822 20237 tablet_service.cc:1467] Processing CreateTablet for tablet cd4569304ba946d0b6a348f665669d8a (DEFAULT_TABLE table=test-table [id=59287bdf11bb4470a474c2c43fe3ea47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:06:22.959578 20368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4569304ba946d0b6a348f665669d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:22.959729 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4569304ba946d0b6a348f665669d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:22.966593 20583 tablet_bootstrap.cc:492] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: Bootstrap starting.
I20260927 15:06:22.966777 20582 tablet_bootstrap.cc:492] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Bootstrap starting.
I20260927 15:06:22.968573 20583 tablet_bootstrap.cc:654] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:22.968597 20582 tablet_bootstrap.cc:654] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:22.969291 20583 log.cc:824] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:22.969292 20582 log.cc:824] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:22.969985 20584 tablet_bootstrap.cc:492] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Bootstrap starting.
I20260927 15:06:22.971544 20583 tablet_bootstrap.cc:492] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: No bootstrap required, opened a new log
I20260927 15:06:22.971544 20582 tablet_bootstrap.cc:492] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: No bootstrap required, opened a new log
I20260927 15:06:22.971741 20583 ts_tablet_manager.cc:1397] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:06:22.971741 20582 ts_tablet_manager.cc:1397] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:06:22.971993 20584 tablet_bootstrap.cc:654] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:22.972754 20584 log.cc:824] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:22.974577 20584 tablet_bootstrap.cc:492] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: No bootstrap required, opened a new log
I20260927 15:06:22.974768 20584 ts_tablet_manager.cc:1397] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:06:22.976308 20583 raft_consensus.cc:348] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:22.976308 20582 raft_consensus.cc:348] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:22.976585 20582 raft_consensus.cc:374] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:22.976585 20583 raft_consensus.cc:374] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:22.976657 20583 raft_consensus.cc:729] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb26eeb5f2440ba9b2ef3ab8fdab296, State: Initialized, Role: FOLLOWER
I20260927 15:06:22.976649 20582 raft_consensus.cc:729] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a03cdb5475445c5ad5ab31470c5fff9, State: Initialized, Role: FOLLOWER
I20260927 15:06:22.977128 20582 consensus_queue.cc:260] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:22.977128 20583 consensus_queue.cc:260] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:22.978204 20583 ts_tablet_manager.cc:1428] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 15:06:22.978193 20582 ts_tablet_manager.cc:1428] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:06:22.978536 20303 heartbeater.cc:503] Master 127.0.98.62:41417 was elected leader, sending a full tablet report...
I20260927 15:06:22.978551 20584 raft_consensus.cc:348] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:22.978796 20584 raft_consensus.cc:374] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:22.978832 20434 heartbeater.cc:503] Master 127.0.98.62:41417 was elected leader, sending a full tablet report...
I20260927 15:06:22.978845 20584 raft_consensus.cc:729] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab944321633b489ab2f81ab1e7870ee4, State: Initialized, Role: FOLLOWER
I20260927 15:06:22.979255 20584 consensus_queue.cc:260] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:22.980020 20565 heartbeater.cc:503] Master 127.0.98.62:41417 was elected leader, sending a full tablet report...
I20260927 15:06:22.980199 20584 ts_tablet_manager.cc:1428] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 15:06:23.000980 20304 tablet.cc:2369] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:06:23.157016 20566 tablet.cc:2369] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:23.193850 20590 raft_consensus.cc:482] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:06:23.194108 20590 raft_consensus.cc:504] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:23.195449 20590 leader_election.cc:288] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a03cdb5475445c5ad5ab31470c5fff9 (127.0.98.2:35781), ceb26eeb5f2440ba9b2ef3ab8fdab296 (127.0.98.1:33397)
I20260927 15:06:23.199476 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:23.195639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35781 (local address 127.0.98.3:47193)
0927 15:06:23.195810 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:23.195828 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:23.195934 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:23.196245 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:23.196247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:23.196264 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:23.196419 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:23.196426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.197274 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.197278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:23.198294 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:23.198304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.199032 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.199036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:23.199138 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:23.199164 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:06:23.199247 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:06:23.199323 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 15:06:23.199555 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:23.195865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47193 (local address 127.0.98.2:35781)
0927 15:06:23.195990 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:23.195994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:23.196011 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:06:23.196074 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:23.196077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:23.196139 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:23.196230 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:23.196546 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.197172 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.198408 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.198908 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.198997 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:23.199048 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:23.199303 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 15:06:23.199415 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:06:23.199966 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:23.195865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48883 (local address 127.0.98.1:33397)
0927 15:06:23.195990 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:23.195994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:23.196373 (+ 379us) server_negotiation.cc:407] Connection header received
0927 15:06:23.196439 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:23.196442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:23.196505 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:23.196582 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:06:23.196865 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.197465 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.198749 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.199271 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.199374 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:23.199428 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:23.199708 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 15:06:23.199813 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:23.200368 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:23.195759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33397 (local address 127.0.98.3:48883)
0927 15:06:23.196210 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:23.196234 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:23.196329 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:23.196602 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:23.196604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:23.196621 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:06:23.196739 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:23.196746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.197575 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.197578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:23.198626 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:23.198636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.199403 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.199407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:23.199515 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:23.199540 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:06:23.200130 (+ 590us) client_negotiation.cc:240] Negotiation successful
0927 15:06:23.200258 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":140,"threads_started":1}
I20260927 15:06:23.200469 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4569304ba946d0b6a348f665669d8a" candidate_uuid: "ab944321633b489ab2f81ab1e7870ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" is_pre_election: true
I20260927 15:06:23.200834 20388 raft_consensus.cc:2454] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab944321633b489ab2f81ab1e7870ee4 in term 0.
I20260927 15:06:23.201041 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4569304ba946d0b6a348f665669d8a" candidate_uuid: "ab944321633b489ab2f81ab1e7870ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" is_pre_election: true
I20260927 15:06:23.201351 20257 raft_consensus.cc:2454] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab944321633b489ab2f81ab1e7870ee4 in term 0.
I20260927 15:06:23.201357 20452 leader_election.cc:302] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a03cdb5475445c5ad5ab31470c5fff9, ab944321633b489ab2f81ab1e7870ee4; no voters:
I20260927 15:06:23.201610 20590 raft_consensus.cc:2781] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:06:23.201709 20590 raft_consensus.cc:482] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:06:23.201761 20590 raft_consensus.cc:3037] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:23.202852 20590 raft_consensus.cc:504] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:23.203241 20590 leader_election.cc:288] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [CANDIDATE]: Term 1 election: Requested vote from peers 3a03cdb5475445c5ad5ab31470c5fff9 (127.0.98.2:35781), ceb26eeb5f2440ba9b2ef3ab8fdab296 (127.0.98.1:33397)
I20260927 15:06:23.203565 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4569304ba946d0b6a348f665669d8a" candidate_uuid: "ab944321633b489ab2f81ab1e7870ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a03cdb5475445c5ad5ab31470c5fff9"
I20260927 15:06:23.203709 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4569304ba946d0b6a348f665669d8a" candidate_uuid: "ab944321633b489ab2f81ab1e7870ee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296"
I20260927 15:06:23.203742 20388 raft_consensus.cc:3037] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:23.203899 20257 raft_consensus.cc:3037] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:23.204890 20388 raft_consensus.cc:2454] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab944321633b489ab2f81ab1e7870ee4 in term 1.
I20260927 15:06:23.205035 20257 raft_consensus.cc:2454] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab944321633b489ab2f81ab1e7870ee4 in term 1.
I20260927 15:06:23.205206 20452 leader_election.cc:302] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a03cdb5475445c5ad5ab31470c5fff9, ab944321633b489ab2f81ab1e7870ee4; no voters:
I20260927 15:06:23.205408 20590 raft_consensus.cc:2781] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:23.205648 20590 raft_consensus.cc:686] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 1 LEADER]: Becoming Leader. State: Replica: ab944321633b489ab2f81ab1e7870ee4, State: Running, Role: LEADER
I20260927 15:06:23.206050 20590 consensus_queue.cc:237] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } }
I20260927 15:06:23.209190 20114 catalog_manager.cc:5495] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab944321633b489ab2f81ab1e7870ee4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ab944321633b489ab2f81ab1e7870ee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab944321633b489ab2f81ab1e7870ee4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 } health_report { overall_health: UNKNOWN } } }
W20260927 15:06:23.215559 20435 tablet.cc:2369] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:06:23.248122 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:23.244405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.0.98.3:36375)
0927 15:06:23.244541 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:23.244545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:23.244560 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:06:23.244748 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:23.244750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:23.244805 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:23.244918 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:23.245173 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.245696 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.246581 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:23.246799 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:23.246879 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:23.247244 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:23.247657 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:23.247756 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:23.247879 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 15:06:23.247922 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:06:23.253888 20257 raft_consensus.cc:1261] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 1 FOLLOWER]: Refusing update from remote peer ab944321633b489ab2f81ab1e7870ee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:23.253890 20388 raft_consensus.cc:1261] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 1 FOLLOWER]: Refusing update from remote peer ab944321633b489ab2f81ab1e7870ee4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:23.254580 20592 consensus_queue.cc:1035] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a03cdb5475445c5ad5ab31470c5fff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:23.254773 20590 consensus_queue.cc:1035] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb26eeb5f2440ba9b2ef3ab8fdab296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:23.264307 20596 mvcc.cc:205] Tried to move back new op lower bound from 7333976405001224192 to 7333976404814802944. Current Snapshot: MvccSnapshot[applied={T|T < 7333976405001224192}]
I20260927 15:06:23.270643 20115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:23.270792 20115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:06:23.273351 20115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c677dc9f26840338996bf11accb20cb: Sending DeleteTablet for 3 replicas of tablet cd4569304ba946d0b6a348f665669d8a
I20260927 15:06:23.274224 20368 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4569304ba946d0b6a348f665669d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:23 UTC) from {username='slave'} at 127.0.0.1:55434
I20260927 15:06:23.274339 20237 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4569304ba946d0b6a348f665669d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:23 UTC) from {username='slave'} at 127.0.0.1:42172
I20260927 15:06:23.274453 20499 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4569304ba946d0b6a348f665669d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:06:23 UTC) from {username='slave'} at 127.0.0.1:51328
I20260927 15:06:23.274830 20606 tablet_replica.cc:330] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296: stopping tablet replica
I20260927 15:06:23.274849 20607 tablet_replica.cc:330] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: stopping tablet replica
I20260927 15:06:23.275144 20607 raft_consensus.cc:2229] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:06:23.275221 20606 raft_consensus.cc:2229] T cd4569304ba946d0b6a348f665669d8a P ceb26eeb5f2440ba9b2ef3ab8fdab296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:23.275564 20607 raft_consensus.cc:2258] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:23.275715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20175
I20260927 15:06:23.275719 20605 tablet_replica.cc:330] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: stopping tablet replica
I20260927 15:06:23.276436 20605 raft_consensus.cc:2229] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:06:23.276806 20605 raft_consensus.cc:2258] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:06:23.278424 20605 ts_tablet_manager.cc:1905] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:23.279376 20607 ts_tablet_manager.cc:1905] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:06:23.281334 20605 ts_tablet_manager.cc:1918] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:23.281510 20605 log.cc:1196] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-1/wal/wals/cd4569304ba946d0b6a348f665669d8a
I20260927 15:06:23.281931 20605 ts_tablet_manager.cc:1939] T cd4569304ba946d0b6a348f665669d8a P 3a03cdb5475445c5ad5ab31470c5fff9: Deleting consensus metadata
I20260927 15:06:23.282194 20607 ts_tablet_manager.cc:1918] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:06:23.282390 20607 log.cc:1196] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Deleting WAL directory at /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521328452215-392-0/minicluster-data/ts-2/wal/wals/cd4569304ba946d0b6a348f665669d8a
I20260927 15:06:23.282706 20607 ts_tablet_manager.cc:1939] T cd4569304ba946d0b6a348f665669d8a P ab944321633b489ab2f81ab1e7870ee4: Deleting consensus metadata
I20260927 15:06:23.283459 20099 catalog_manager.cc:4841] TS 3a03cdb5475445c5ad5ab31470c5fff9 (127.0.98.2:35781): tablet cd4569304ba946d0b6a348f665669d8a (table test-table [id=59287bdf11bb4470a474c2c43fe3ea47]) successfully deleted
W20260927 15:06:23.283589 20101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33397 (error 108)
I20260927 15:06:23.284271 20102 catalog_manager.cc:4841] TS ab944321633b489ab2f81ab1e7870ee4 (127.0.98.3:36375): tablet cd4569304ba946d0b6a348f665669d8a (table test-table [id=59287bdf11bb4470a474c2c43fe3ea47]) successfully deleted
I20260927 15:06:23.284580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20306
I20260927 15:06:23.284883 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:23.284582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33397 (local address 127.0.0.1:42188)
0927 15:06:23.284730 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:23.284803 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 15:06:23.286651 20101 catalog_manager.cc:4584] TS ceb26eeb5f2440ba9b2ef3ab8fdab296 (127.0.98.1:33397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd4569304ba946d0b6a348f665669d8a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33397: connect: Connection refused (error 111)
I20260927 15:06:23.290645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20437
I20260927 15:06:23.296125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20084
2026-09-27T15:06:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1280 ms)
[----------] 6 tests from EncryptionPolicyTest (11004 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:23 dist-test-slave-ml7t krb5kdc[20611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:23 dist-test-slave-ml7t krb5kdc[20611](info): set up 2 sockets
Sep 27 15:06:23 dist-test-slave-ml7t krb5kdc[20611](info): commencing operation
krb5kdc: starting...
W20260927 15:06:24.337927 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:24 dist-test-slave-ml7t krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:24.457800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:24.567350 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:24.567612 20624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:24.567654 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:24.571720 20624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:24.571780 20624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:24.571800 20624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:24.571816 20624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:24.571831 20624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:24.577370 20624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:24.578450 20624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:24.579474 20624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:24.588073 20630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:24.588248 20624 server_base.cc:1034] running on GCE node
W20260927 15:06:24.588079 20629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:24.588088 20632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:24.588857 20624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:24.590034 20624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:24.591225 20624 hybrid_clock.cc:648] HybridClock initialized: now 1790521584591216 us; error 27 us; skew 500 ppm
Sep 27 15:06:24 dist-test-slave-ml7t krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:24.593905 20624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:24.594758 20624 webserver.cc:460] Webserver started at http://127.0.0.1:38859/ using document root <none> and password file <none>
I20260927 15:06:24.595299 20624 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:24.595409 20624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:24.595635 20624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:24.597586 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "d585b35df1834daf814859baba8f0969"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-ml7t"
server_key: "4bd2fdcef01c8697f88835b64a76adbf"
server_key_iv: "5b35dcbfa699965ebd5ea72fb792fe20"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.598193 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "d585b35df1834daf814859baba8f0969"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-ml7t"
server_key: "4bd2fdcef01c8697f88835b64a76adbf"
server_key_iv: "5b35dcbfa699965ebd5ea72fb792fe20"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.601624 20624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:24.603986 20639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:24.605196 20624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:24.605474 20624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "d585b35df1834daf814859baba8f0969"
format_stamp: "Formatted at 2026-09-27 15:06:24 on dist-test-slave-ml7t"
server_key: "4bd2fdcef01c8697f88835b64a76adbf"
server_key_iv: "5b35dcbfa699965ebd5ea72fb792fe20"
server_key_version: "encryptionkey@0"
I20260927 15:06:24.605616 20624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:24.624810 20624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:24.628213 20624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:24.628448 20624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:24.635526 20624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37357
I20260927 15:06:24.635551 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37357 every 8 connection(s)
I20260927 15:06:24.636492 20624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:24.639248 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:24.642679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20624
I20260927 15:06:24.642805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:24.643087 392 external_mini_cluster.cc:1428] Setting key 61f8d7e4da36acbdd2a21f9c605c8795
I20260927 15:06:24.645113 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969: Bootstrap starting.
I20260927 15:06:24.647244 20692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:24.647954 20692 log.cc:824] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:24.649677 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969: No bootstrap required, opened a new log
Sep 27 15:06:24 dist-test-slave-ml7t krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:06:24.653347 20692 raft_consensus.cc:348] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d585b35df1834daf814859baba8f0969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260927 15:06:24.653582 20692 raft_consensus.cc:374] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:24.653662 20692 raft_consensus.cc:729] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d585b35df1834daf814859baba8f0969, State: Initialized, Role: FOLLOWER
I20260927 15:06:24.654107 20692 consensus_queue.cc:260] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d585b35df1834daf814859baba8f0969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260927 15:06:24.654253 20692 raft_consensus.cc:388] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:24.654320 20692 raft_consensus.cc:482] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:24.654448 20692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:24.655280 20692 raft_consensus.cc:504] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d585b35df1834daf814859baba8f0969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260927 15:06:24.655603 20692 leader_election.cc:302] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d585b35df1834daf814859baba8f0969; no voters:
I20260927 15:06:24.655938 20692 leader_election.cc:288] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:24.656001 20697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:24.656236 20697 raft_consensus.cc:686] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [term 1 LEADER]: Becoming Leader. State: Replica: d585b35df1834daf814859baba8f0969, State: Running, Role: LEADER
I20260927 15:06:24.656549 20697 consensus_queue.cc:237] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d585b35df1834daf814859baba8f0969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260927 15:06:24.657181 20692 sys_catalog.cc:564] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:24.657445 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:24.644522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34113 (local address 127.0.0.1:37357)
0927 15:06:24.644952 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:24.644966 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:24.645000 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:06:24.645656 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:24.645674 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:24.646073 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:24.646392 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:24.647184 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.648258 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.649100 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.649555 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.649716 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:24.652215 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:24.652220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:24.652232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:24.652256 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:24.654148 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.654691 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.654696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.654704 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.654814 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.655159 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.655163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.655164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.655598 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:24.655822 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:24.656234 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 15:06:24.656534 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
I20260927 15:06:24.657716 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d585b35df1834daf814859baba8f0969. Latest consensus state: current_term: 1 leader_uuid: "d585b35df1834daf814859baba8f0969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d585b35df1834daf814859baba8f0969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } } }
I20260927 15:06:24.657845 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:24.658293 20702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:24.658433 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d585b35df1834daf814859baba8f0969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d585b35df1834daf814859baba8f0969" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } } }
I20260927 15:06:24.658576 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:24.660665 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:24.661396 20702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:24.668182 20702 catalog_manager.cc:1347] Generated new cluster ID: 366f3923ad3e4125ba40e74a2f91c0ab
I20260927 15:06:24.668252 20702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:24.671649 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:24.663144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33565 (local address 127.0.0.1:37357)
0927 15:06:24.663264 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:24.663268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:24.663306 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:06:24.663451 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:24.663454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:24.663502 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:24.663585 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:24.664304 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.664824 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.665754 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:24.666162 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:24.666295 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:24.667974 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:24.667977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:24.667980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:24.668007 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:24.669842 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.670359 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.670363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.670365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.670455 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:24.670772 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:24.670775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:24.670777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:24.670994 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:24.671143 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:24.671395 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 15:06:24.671491 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:06:24.688474 20702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:24.689646 20702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:24.697991 20702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d585b35df1834daf814859baba8f0969: Generated new TSK 0
I20260927 15:06:24.698844 20702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:24.703578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20624
W20260927 15:06:24.708483 20609 connection.cc:535] client connection to 127.0.0.1:37357 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37357): Connection reset by peer (error 104)
2026-09-27T15:06:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:24 dist-test-slave-ml7t krb5kdc[20723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:24 dist-test-slave-ml7t krb5kdc[20723](info): set up 2 sockets
Sep 27 15:06:24 dist-test-slave-ml7t krb5kdc[20723](info): commencing operation
krb5kdc: starting...
W20260927 15:06:25.738019 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:25 dist-test-slave-ml7t krb5kdc[20723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:25.862316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:25.973133 20736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:25.973395 20736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:25.973436 20736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:25.977474 20736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:25.977537 20736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:25.977556 20736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:25.977573 20736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:25.977587 20736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:25.983047 20736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:25.984138 20736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:25.985173 20736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:25.993696 20744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:25.993760 20736 server_base.cc:1034] running on GCE node
W20260927 15:06:25.993700 20742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:25.993669 20741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:25.994587 20736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:25.995631 20736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:25.996826 20736 hybrid_clock.cc:648] HybridClock initialized: now 1790521585996805 us; error 27 us; skew 500 ppm
Sep 27 15:06:25 dist-test-slave-ml7t krb5kdc[20723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:25.999603 20736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:26.000499 20736 webserver.cc:460] Webserver started at http://127.0.0.1:37719/ using document root <none> and password file <none>
I20260927 15:06:26.001041 20736 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:26.001101 20736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:26.001354 20736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:26.003471 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "0c0a7856bde24739a209be595de41711"
format_stamp: "Formatted at 2026-09-27 15:06:25 on dist-test-slave-ml7t"
server_key: "d1c2ac22ff6ca8721e01efb0eeef541c"
server_key_iv: "f3d5c3c2c930d4a018cc1244d57d4f6e"
server_key_version: "encryptionkey@0"
I20260927 15:06:26.004066 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c0a7856bde24739a209be595de41711"
format_stamp: "Formatted at 2026-09-27 15:06:25 on dist-test-slave-ml7t"
server_key: "d1c2ac22ff6ca8721e01efb0eeef541c"
server_key_iv: "f3d5c3c2c930d4a018cc1244d57d4f6e"
server_key_version: "encryptionkey@0"
I20260927 15:06:26.008303 20736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 15:06:26.011181 20751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:26.012694 20736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:06:26.012873 20736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "0c0a7856bde24739a209be595de41711"
format_stamp: "Formatted at 2026-09-27 15:06:25 on dist-test-slave-ml7t"
server_key: "d1c2ac22ff6ca8721e01efb0eeef541c"
server_key_iv: "f3d5c3c2c930d4a018cc1244d57d4f6e"
server_key_version: "encryptionkey@0"
I20260927 15:06:26.013013 20736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:26.041235 20736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:26.044576 20736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:26.044809 20736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:26.052292 20736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39539
I20260927 15:06:26.052636 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39539 every 8 connection(s)
I20260927 15:06:26.053320 20736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:26.056138 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:26.059080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20736
I20260927 15:06:26.059231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:26.059504 392 external_mini_cluster.cc:1428] Setting key fbe88608d5468258342bc59ac4c57e36
I20260927 15:06:26.061957 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711: Bootstrap starting.
I20260927 15:06:26.064149 20804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:26.065119 20804 log.cc:824] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:26.067252 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711: No bootstrap required, opened a new log
I20260927 15:06:26.069271 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:26.060894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47267 (local address 127.0.0.1:39539)
0927 15:06:26.061343 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:26.061358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:26.061393 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:06:26.062129 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:26.062150 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:26.062533 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:26.062870 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:26.063284 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:26.064471 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:26.065729 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:26.066332 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:26.066451 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:26.067436 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:26.067441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:26.067454 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:26.067821 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:26.067880 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:26.067902 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:26.068005 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:26.068193 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 15:06:26.068380 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260927 15:06:26.070396 20804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0a7856bde24739a209be595de41711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260927 15:06:26.070600 20804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:26.070658 20804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0a7856bde24739a209be595de41711, State: Initialized, Role: FOLLOWER
I20260927 15:06:26.071125 20804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0a7856bde24739a209be595de41711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260927 15:06:26.071344 20804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:26.071447 20804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:26.071557 20804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:26.072393 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:26.072733 20804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0a7856bde24739a209be595de41711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260927 15:06:26.073073 20804 leader_election.cc:302] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c0a7856bde24739a209be595de41711; no voters:
I20260927 15:06:26.073406 20804 leader_election.cc:288] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:26.073485 20809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:26.073671 20809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [term 1 LEADER]: Becoming Leader. State: Replica: 0c0a7856bde24739a209be595de41711, State: Running, Role: LEADER
I20260927 15:06:26.074003 20809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0a7856bde24739a209be595de41711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260927 15:06:26.074702 20804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:26.075170 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c0a7856bde24739a209be595de41711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0a7856bde24739a209be595de41711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } } }
I20260927 15:06:26.075304 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:26.075512 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c0a7856bde24739a209be595de41711. Latest consensus state: current_term: 1 leader_uuid: "0c0a7856bde24739a209be595de41711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0a7856bde24739a209be595de41711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } } }
I20260927 15:06:26.075613 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:26.076005 20819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:26.079197 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:26.074505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32963 (local address 127.0.0.1:39539)
0927 15:06:26.074609 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:26.074613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:26.074836 (+ 223us) server_negotiation.cc:407] Connection header received
0927 15:06:26.075014 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:26.075017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:26.075078 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:26.075159 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:26.075449 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:26.076128 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:26.077346 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:26.077586 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:26.077696 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:26.078618 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:26.078623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:26.078629 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:26.078643 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:26.078669 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:26.078683 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:26.078752 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:26.079000 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:06:26.079049 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:06:26.079329 20819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:26.084829 20819 catalog_manager.cc:1347] Generated new cluster ID: a426397b22eb40f5b103155985c86cb8
I20260927 15:06:26.084905 20819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:26.096264 20819 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:26.097174 20819 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:26.109259 20819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c0a7856bde24739a209be595de41711: Generated new TSK 0
I20260927 15:06:26.110013 20819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:26.114696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20736
W20260927 15:06:26.120106 20721 connection.cc:535] client connection to 127.0.0.1:39539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39539): Connection reset by peer (error 104)
2026-09-27T15:06:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:26 dist-test-slave-ml7t krb5kdc[20835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:26 dist-test-slave-ml7t krb5kdc[20835](info): set up 2 sockets
Sep 27 15:06:26 dist-test-slave-ml7t krb5kdc[20835](info): commencing operation
krb5kdc: starting...
W20260927 15:06:27.162039 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:27 dist-test-slave-ml7t krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:27.285760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:27.392906 20848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:27.393184 20848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:27.393262 20848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:27.397230 20848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:27.397315 20848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:27.397351 20848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:27.397405 20848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:27.397442 20848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:27.403092 20848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:27.404184 20848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:27.405220 20848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:27.413625 20853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:27.413635 20856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:27.413697 20854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:27.413997 20848 server_base.cc:1034] running on GCE node
I20260927 15:06:27.414537 20848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:27.415629 20848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:27.416815 20848 hybrid_clock.cc:648] HybridClock initialized: now 1790521587416801 us; error 28 us; skew 500 ppm
Sep 27 15:06:27 dist-test-slave-ml7t krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:27.419685 20848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:27.420596 20848 webserver.cc:460] Webserver started at http://127.0.0.1:45367/ using document root <none> and password file <none>
I20260927 15:06:27.421187 20848 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:27.421277 20848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:27.421485 20848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:27.423518 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "217bb113a82e489f893bfd9634dfec85"
format_stamp: "Formatted at 2026-09-27 15:06:27 on dist-test-slave-ml7t"
server_key: "ad9cbe421da3d1f1543be1a3563d81f7"
server_key_iv: "e310e3fa91a1243173534209decfb171"
server_key_version: "encryptionkey@0"
I20260927 15:06:27.424088 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "217bb113a82e489f893bfd9634dfec85"
format_stamp: "Formatted at 2026-09-27 15:06:27 on dist-test-slave-ml7t"
server_key: "ad9cbe421da3d1f1543be1a3563d81f7"
server_key_iv: "e310e3fa91a1243173534209decfb171"
server_key_version: "encryptionkey@0"
I20260927 15:06:27.427727 20848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:06:27.430148 20863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:27.431604 20848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:27.431747 20848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "217bb113a82e489f893bfd9634dfec85"
format_stamp: "Formatted at 2026-09-27 15:06:27 on dist-test-slave-ml7t"
server_key: "ad9cbe421da3d1f1543be1a3563d81f7"
server_key_iv: "e310e3fa91a1243173534209decfb171"
server_key_version: "encryptionkey@0"
I20260927 15:06:27.431859 20848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:27.436403 20848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:27.439622 20848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:27.439816 20848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:27.446684 20848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36929
I20260927 15:06:27.446712 20915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36929 every 8 connection(s)
I20260927 15:06:27.447564 20848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:27.450384 20916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:27.451822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20848
I20260927 15:06:27.452001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:27.452381 392 external_mini_cluster.cc:1428] Setting key 87b694683789fbdb7e11cb897c17abdd
I20260927 15:06:27.457468 20916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85: Bootstrap starting.
I20260927 15:06:27.459822 20916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:27.460765 20916 log.cc:824] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:27.461697 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:27.453902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57687 (local address 127.0.0.1:36929)
0927 15:06:27.454340 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:27.454361 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:27.454407 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:06:27.455306 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:27.455333 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:27.455722 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:27.456108 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:27.460225 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:27.460232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:27.460248 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:27.460263 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:27.460332 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:27.460361 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:27.460499 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:27.460580 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:06:27.460750 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
I20260927 15:06:27.462957 20916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85: No bootstrap required, opened a new log
I20260927 15:06:27.464706 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:27.467435 20916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bb113a82e489f893bfd9634dfec85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 15:06:27.467662 20916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:27.467787 20916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217bb113a82e489f893bfd9634dfec85, State: Initialized, Role: FOLLOWER
I20260927 15:06:27.468328 20916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bb113a82e489f893bfd9634dfec85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 15:06:27.468447 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:27.466764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57091 (local address 127.0.0.1:36929)
0927 15:06:27.466892 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:27.466898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:27.467118 (+ 220us) server_negotiation.cc:407] Connection header received
0927 15:06:27.467199 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:27.467202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:27.467238 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:27.467322 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:27.468054 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:27.468057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:27.468060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:27.468072 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:27.468100 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:27.468113 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:27.468221 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:27.468256 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 15:06:27.468293 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:06:27.468515 20916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:27.468580 20916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:27.468719 20916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:27.469887 20916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bb113a82e489f893bfd9634dfec85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 15:06:27.470197 20916 leader_election.cc:302] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217bb113a82e489f893bfd9634dfec85; no voters:
I20260927 15:06:27.470461 20916 leader_election.cc:288] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:27.470625 20926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:27.470877 20926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [term 1 LEADER]: Becoming Leader. State: Replica: 217bb113a82e489f893bfd9634dfec85, State: Running, Role: LEADER
I20260927 15:06:27.471192 20926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bb113a82e489f893bfd9634dfec85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 15:06:27.471796 20916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:27.472937 20927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "217bb113a82e489f893bfd9634dfec85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bb113a82e489f893bfd9634dfec85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } } }
I20260927 15:06:27.472935 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217bb113a82e489f893bfd9634dfec85. Latest consensus state: current_term: 1 leader_uuid: "217bb113a82e489f893bfd9634dfec85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bb113a82e489f893bfd9634dfec85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } } }
I20260927 15:06:27.473081 20927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:27.473081 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:27.473512 20935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:27.476433 20935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:27.482120 20935 catalog_manager.cc:1347] Generated new cluster ID: 0dc0b068bc694b509167df5adeb89f84
I20260927 15:06:27.482187 20935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:27.499804 20935 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:27.500157 20935 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:27.510499 20935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 217bb113a82e489f893bfd9634dfec85: Generated new TSK 0
I20260927 15:06:27.511369 20935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:27.514371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20848
2026-09-27T15:06:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:27 dist-test-slave-ml7t krb5kdc[20947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:27 dist-test-slave-ml7t krb5kdc[20947](info): set up 2 sockets
Sep 27 15:06:27 dist-test-slave-ml7t krb5kdc[20947](info): commencing operation
krb5kdc: starting...
W20260927 15:06:28.561969 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:28 dist-test-slave-ml7t krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:28.689518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:28.800832 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:28.801118 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:28.801194 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:28.805542 20960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:28.805646 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:28.805691 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:28.805723 20960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:28.805781 20960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:28.811789 20960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:28.812914 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:28.814004 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:28.822626 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:28.822630 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:28.822664 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:28.823067 20960 server_base.cc:1034] running on GCE node
I20260927 15:06:28.823658 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:28.824834 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:28.826057 20960 hybrid_clock.cc:648] HybridClock initialized: now 1790521588826037 us; error 35 us; skew 500 ppm
Sep 27 15:06:28 dist-test-slave-ml7t krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:28.829106 20960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:28.830165 20960 webserver.cc:460] Webserver started at http://127.0.0.1:33689/ using document root <none> and password file <none>
I20260927 15:06:28.830757 20960 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:28.830855 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:28.831103 20960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:28.833192 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "67b4d1b03c7e4b39874c8a09445192de"
format_stamp: "Formatted at 2026-09-27 15:06:28 on dist-test-slave-ml7t"
server_key: "0a1f832ecb24f23bf280d8f44d17a910"
server_key_iv: "e8cff9f06c80dfbd486497493de3d074"
server_key_version: "encryptionkey@0"
I20260927 15:06:28.833863 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "67b4d1b03c7e4b39874c8a09445192de"
format_stamp: "Formatted at 2026-09-27 15:06:28 on dist-test-slave-ml7t"
server_key: "0a1f832ecb24f23bf280d8f44d17a910"
server_key_iv: "e8cff9f06c80dfbd486497493de3d074"
server_key_version: "encryptionkey@0"
I20260927 15:06:28.837652 20960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:06:28.840116 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:28.841316 20960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:28.841471 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "67b4d1b03c7e4b39874c8a09445192de"
format_stamp: "Formatted at 2026-09-27 15:06:28 on dist-test-slave-ml7t"
server_key: "0a1f832ecb24f23bf280d8f44d17a910"
server_key_iv: "e8cff9f06c80dfbd486497493de3d074"
server_key_version: "encryptionkey@0"
I20260927 15:06:28.841589 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:28.858956 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:28.862043 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:28.862274 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:28.869581 20960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36825
I20260927 15:06:28.869637 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36825 every 8 connection(s)
I20260927 15:06:28.870656 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:28.873451 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:28.876078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 20960
I20260927 15:06:28.876214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:28.876493 392 external_mini_cluster.cc:1428] Setting key 2035a904e10ed811d8aaf2de673d833a
I20260927 15:06:28.879920 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de: Bootstrap starting.
I20260927 15:06:28.882647 21028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:28.883448 21028 log.cc:824] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:28.885412 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de: No bootstrap required, opened a new log
Sep 27 15:06:28 dist-test-slave-ml7t krb5kdc[20947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:06:28.889750 21028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b4d1b03c7e4b39874c8a09445192de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:06:28.890012 21028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:28.890082 21028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b4d1b03c7e4b39874c8a09445192de, State: Initialized, Role: FOLLOWER
I20260927 15:06:28.890623 21028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b4d1b03c7e4b39874c8a09445192de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:06:28.890829 21028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:28.890900 21028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:28.891003 21028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:28.892202 21028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b4d1b03c7e4b39874c8a09445192de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:06:28.892545 21028 leader_election.cc:302] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b4d1b03c7e4b39874c8a09445192de; no voters:
I20260927 15:06:28.892879 21028 leader_election.cc:288] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:28.893038 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:28.877996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52819 (local address 127.0.0.1:36825)
0927 15:06:28.878421 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:28.878441 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:28.878486 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:06:28.879296 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:28.879319 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:28.879743 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:28.880125 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:28.881006 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.882519 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.883491 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.884089 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.884284 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:28.886920 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:28.886928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:28.886947 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:28.886983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:28.889656 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.890278 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.890283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.890288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.890406 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.890694 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.890696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.890698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.891262 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:28.891416 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:28.891766 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 15:06:28.892100 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260927 15:06:28.893013 21033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:28.893301 21033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [term 1 LEADER]: Becoming Leader. State: Replica: 67b4d1b03c7e4b39874c8a09445192de, State: Running, Role: LEADER
I20260927 15:06:28.893600 21033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b4d1b03c7e4b39874c8a09445192de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:06:28.894166 21028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:28.894773 21034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b4d1b03c7e4b39874c8a09445192de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b4d1b03c7e4b39874c8a09445192de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260927 15:06:28.894894 21034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:28.895133 21035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b4d1b03c7e4b39874c8a09445192de. Latest consensus state: current_term: 1 leader_uuid: "67b4d1b03c7e4b39874c8a09445192de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b4d1b03c7e4b39874c8a09445192de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260927 15:06:28.895213 21035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:28.895823 21039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:28.896457 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:28.899379 21039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:28.905874 21039 catalog_manager.cc:1347] Generated new cluster ID: 9ad7b1f085d84805999eb35339acef3d
I20260927 15:06:28.905946 21039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:28.908063 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:28.898507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38735 (local address 127.0.0.1:36825)
0927 15:06:28.898588 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:28.898592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:28.898985 (+ 393us) server_negotiation.cc:407] Connection header received
0927 15:06:28.899110 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:28.899113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:28.899162 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:28.899302 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:28.900302 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.900884 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.902046 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:28.902423 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:28.902540 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:28.904103 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:28.904107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:28.904110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:28.904134 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:28.905941 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.906463 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.906467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.906469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.906553 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:28.906848 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:28.906853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:28.906856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:28.907064 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:28.907182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:28.907804 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 15:06:28.907915 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 15:06:28.929180 21039 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:28.930207 21039 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:28.940557 21039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b4d1b03c7e4b39874c8a09445192de: Generated new TSK 0
I20260927 15:06:28.941458 21039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:28.952173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 20960
2026-09-27T15:06:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:29 dist-test-slave-ml7t krb5kdc[21059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:29 dist-test-slave-ml7t krb5kdc[21059](info): set up 2 sockets
Sep 27 15:06:29 dist-test-slave-ml7t krb5kdc[21059](info): commencing operation
krb5kdc: starting...
W20260927 15:06:29.990283 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:30 dist-test-slave-ml7t krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:30.117396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:30.231104 21072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:30.231443 21072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:30.231528 21072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:30.235684 21072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:30.235776 21072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:30.235816 21072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:30.235878 21072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:30.235918 21072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:30.241272 21072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:30.242621 21072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:30.243631 21072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:30.252107 21080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:30.252094 21078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:30.252131 21077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:30.252336 21072 server_base.cc:1034] running on GCE node
I20260927 15:06:30.252908 21072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:30.254098 21072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:30.255285 21072 hybrid_clock.cc:648] HybridClock initialized: now 1790521590255267 us; error 39 us; skew 500 ppm
Sep 27 15:06:30 dist-test-slave-ml7t krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:30.258072 21072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:30.258949 21072 webserver.cc:460] Webserver started at http://127.0.0.1:36219/ using document root <none> and password file <none>
I20260927 15:06:30.259517 21072 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:30.259608 21072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:30.259840 21072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:30.261931 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "b048fee572e849aea5a58ee96b069605"
format_stamp: "Formatted at 2026-09-27 15:06:30 on dist-test-slave-ml7t"
server_key: "396685bf234a90c65c6958dd4e6e6b25"
server_key_iv: "4771182a9e7856c1d1d12a696c520b3f"
server_key_version: "encryptionkey@0"
I20260927 15:06:30.262492 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "b048fee572e849aea5a58ee96b069605"
format_stamp: "Formatted at 2026-09-27 15:06:30 on dist-test-slave-ml7t"
server_key: "396685bf234a90c65c6958dd4e6e6b25"
server_key_iv: "4771182a9e7856c1d1d12a696c520b3f"
server_key_version: "encryptionkey@0"
I20260927 15:06:30.266161 21072 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:30.268429 21087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:30.269618 21072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:30.269761 21072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "b048fee572e849aea5a58ee96b069605"
format_stamp: "Formatted at 2026-09-27 15:06:30 on dist-test-slave-ml7t"
server_key: "396685bf234a90c65c6958dd4e6e6b25"
server_key_iv: "4771182a9e7856c1d1d12a696c520b3f"
server_key_version: "encryptionkey@0"
I20260927 15:06:30.269955 21072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:30.282768 21072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:30.286253 21072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:30.286485 21072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:30.294044 21072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34561
I20260927 15:06:30.294095 21139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34561 every 8 connection(s)
I20260927 15:06:30.295043 21072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:30.297828 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:30.303257 21140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605: Bootstrap starting.
I20260927 15:06:30.304193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21072
I20260927 15:06:30.304432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:30.304711 392 external_mini_cluster.cc:1428] Setting key 134caf950960baec764372f76444410f
I20260927 15:06:30.306023 21140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:30.306741 21140 log.cc:824] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:30.308650 21140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605: No bootstrap required, opened a new log
I20260927 15:06:30.312915 21140 raft_consensus.cc:348] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048fee572e849aea5a58ee96b069605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 15:06:30.313161 21140 raft_consensus.cc:374] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:30.313248 21140 raft_consensus.cc:729] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b048fee572e849aea5a58ee96b069605, State: Initialized, Role: FOLLOWER
I20260927 15:06:30.313596 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.305884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53603 (local address 127.0.0.1:34561)
0927 15:06:30.306362 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.306378 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.306413 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:06:30.307069 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.307087 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.307448 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.307721 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:30.308022 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.309033 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.310302 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.310764 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.310884 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.311707 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:30.311712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:30.311724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:30.312115 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:30.312166 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:30.312189 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:30.312266 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.312426 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.312655 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":138,"threads_started":1}
I20260927 15:06:30.313710 21140 consensus_queue.cc:260] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048fee572e849aea5a58ee96b069605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 15:06:30.313890 21140 raft_consensus.cc:388] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:30.313947 21140 raft_consensus.cc:482] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:30.314031 21140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:30.315018 21140 raft_consensus.cc:504] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048fee572e849aea5a58ee96b069605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 15:06:30.315371 21140 leader_election.cc:302] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b048fee572e849aea5a58ee96b069605; no voters:
I20260927 15:06:30.315706 21140 leader_election.cc:288] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:30.315994 21145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:30.316548 21145 raft_consensus.cc:686] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [term 1 LEADER]: Becoming Leader. State: Replica: b048fee572e849aea5a58ee96b069605, State: Running, Role: LEADER
I20260927 15:06:30.316567 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:30.316835 21145 consensus_queue.cc:237] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048fee572e849aea5a58ee96b069605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 15:06:30.316974 21140 sys_catalog.cc:564] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:30.318214 21146 sys_catalog.cc:455] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b048fee572e849aea5a58ee96b069605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048fee572e849aea5a58ee96b069605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } } }
I20260927 15:06:30.318367 21146 sys_catalog.cc:458] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:30.318390 21147 sys_catalog.cc:455] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b048fee572e849aea5a58ee96b069605. Latest consensus state: current_term: 1 leader_uuid: "b048fee572e849aea5a58ee96b069605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048fee572e849aea5a58ee96b069605" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } } }
I20260927 15:06:30.318455 21147 sys_catalog.cc:458] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:30.318818 21158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:30.322846 21158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:30.323725 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:30.319682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34681 (local address 127.0.0.1:34561)
0927 15:06:30.319777 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:30.319782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:30.319796 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:06:30.319835 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:30.319838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:30.319923 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:30.320009 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:30.320303 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.320898 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.322054 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:30.322279 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:30.322389 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:30.323284 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:30.323288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:30.323293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:30.323310 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:30.323340 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:30.323356 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:30.323426 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:30.323547 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 15:06:30.323590 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 15:06:30.328179 21158 catalog_manager.cc:1347] Generated new cluster ID: 565a68f9ae784a4ba7de909807b0f4e4
I20260927 15:06:30.328249 21158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:30.349907 21158 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:30.350898 21158 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:30.360940 21158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b048fee572e849aea5a58ee96b069605: Generated new TSK 0
I20260927 15:06:30.361727 21158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:30.377035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21072
2026-09-27T15:06:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:30 dist-test-slave-ml7t krb5kdc[21171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:30 dist-test-slave-ml7t krb5kdc[21171](info): set up 2 sockets
Sep 27 15:06:30 dist-test-slave-ml7t krb5kdc[21171](info): commencing operation
krb5kdc: starting...
W20260927 15:06:31.406003 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:31 dist-test-slave-ml7t krb5kdc[21171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:31.545594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:31.655210 21184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:31.655504 21184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:31.655550 21184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:31.659650 21184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:31.659713 21184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:31.659732 21184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:31.659749 21184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:31.659765 21184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:31.665242 21184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:31.666437 21184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:31.667512 21184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:06:31.676630 21184 server_base.cc:1034] running on GCE node
W20260927 15:06:31.676537 21189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:31.676541 21192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:31.676554 21190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:31.677413 21184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:31.678627 21184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:31.679801 21184 hybrid_clock.cc:648] HybridClock initialized: now 1790521591679781 us; error 35 us; skew 500 ppm
Sep 27 15:06:31 dist-test-slave-ml7t krb5kdc[21171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:31.682700 21184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:31.683626 21184 webserver.cc:460] Webserver started at http://127.0.0.1:34821/ using document root <none> and password file <none>
I20260927 15:06:31.684180 21184 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:31.684242 21184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:31.684465 21184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:31.686600 21184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "1c4948c1eb7748309d3b0ac7f3d37088"
format_stamp: "Formatted at 2026-09-27 15:06:31 on dist-test-slave-ml7t"
server_key: "de1a5358f7b7d1e8300660a69dc67c87"
server_key_iv: "2e2e05ffaaf5c1c26ea7379fdbf558af"
server_key_version: "encryptionkey@0"
I20260927 15:06:31.687166 21184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c4948c1eb7748309d3b0ac7f3d37088"
format_stamp: "Formatted at 2026-09-27 15:06:31 on dist-test-slave-ml7t"
server_key: "de1a5358f7b7d1e8300660a69dc67c87"
server_key_iv: "2e2e05ffaaf5c1c26ea7379fdbf558af"
server_key_version: "encryptionkey@0"
I20260927 15:06:31.691063 21184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:06:31.693679 21199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:31.694967 21184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:31.695109 21184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "1c4948c1eb7748309d3b0ac7f3d37088"
format_stamp: "Formatted at 2026-09-27 15:06:31 on dist-test-slave-ml7t"
server_key: "de1a5358f7b7d1e8300660a69dc67c87"
server_key_iv: "2e2e05ffaaf5c1c26ea7379fdbf558af"
server_key_version: "encryptionkey@0"
I20260927 15:06:31.695217 21184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:31.699869 21184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:31.702801 21184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:31.702991 21184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:31.710024 21184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41251
I20260927 15:06:31.710057 21251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41251 every 8 connection(s)
I20260927 15:06:31.710953 21184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:31.711921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21184
I20260927 15:06:31.712069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:31.712350 392 external_mini_cluster.cc:1428] Setting key f4307972dd9dfbc21a2c4a8cb7ec56ad
I20260927 15:06:31.714373 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:31.718218 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:31.713889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43261 (local address 127.0.0.1:41251)
0927 15:06:31.714308 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:31.714328 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:31.714373 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:06:31.715263 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:31.715285 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:31.715641 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:31.716026 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:31.716761 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:31.716770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:31.716784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:31.716797 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:31.716864 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:31.716894 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:31.717046 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:31.717122 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 15:06:31.717277 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260927 15:06:31.721093 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:31.721499 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088: Bootstrap starting.
I20260927 15:06:31.724025 21252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:31.724989 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:31.723215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46857 (local address 127.0.0.1:41251)
0927 15:06:31.723347 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:31.723351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:31.723559 (+ 208us) server_negotiation.cc:407] Connection header received
0927 15:06:31.723702 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:31.723704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:31.723733 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:31.723796 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:31.724565 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:31.724568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:31.724570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:31.724580 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:31.724604 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:31.724613 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:31.724680 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:31.724807 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 15:06:31.724837 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:06:31.725029 21252 log.cc:824] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:31.727058 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088: No bootstrap required, opened a new log
I20260927 15:06:31.730530 21252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260927 15:06:31.730774 21252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:31.730819 21252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4948c1eb7748309d3b0ac7f3d37088, State: Initialized, Role: FOLLOWER
I20260927 15:06:31.731194 21252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260927 15:06:31.731304 21252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:31.731348 21252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:31.731412 21252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:31.732327 21252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260927 15:06:31.732627 21252 leader_election.cc:302] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c4948c1eb7748309d3b0ac7f3d37088; no voters:
I20260927 15:06:31.732901 21252 leader_election.cc:288] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:31.733060 21262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:31.733302 21262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [term 1 LEADER]: Becoming Leader. State: Replica: 1c4948c1eb7748309d3b0ac7f3d37088, State: Running, Role: LEADER
I20260927 15:06:31.733625 21262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260927 15:06:31.734230 21252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:31.735368 21264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c4948c1eb7748309d3b0ac7f3d37088. Latest consensus state: current_term: 1 leader_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } } }
I20260927 15:06:31.735503 21264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [sys.catalog]: This master's current role is: LEADER
W20260927 15:06:31.738967 21277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:31.739055 21277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:31.736037 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4948c1eb7748309d3b0ac7f3d37088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } } }
I20260927 15:06:31.739135 21271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:31.739156 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:31.739781 21271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:31.745036 21271 catalog_manager.cc:1347] Generated new cluster ID: 95128815cffa46b8a0aa2c77dce9bdc3
I20260927 15:06:31.745117 21271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:31.761384 21271 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:31.761698 21271 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:31.766059 21271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c4948c1eb7748309d3b0ac7f3d37088: Generated new TSK 0
I20260927 15:06:31.766753 21271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:31.770287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21184
2026-09-27T15:06:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:31 dist-test-slave-ml7t krb5kdc[21283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:31 dist-test-slave-ml7t krb5kdc[21283](info): set up 2 sockets
Sep 27 15:06:31 dist-test-slave-ml7t krb5kdc[21283](info): commencing operation
krb5kdc: starting...
W20260927 15:06:32.810585 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:32 dist-test-slave-ml7t krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:32.941522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:33.052354 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:33.052722 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:33.052814 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:33.057155 21296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:33.057272 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:33.057313 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:33.057353 21296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:33.057389 21296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:33.063053 21296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:33.064266 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:33.065368 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:33.074390 21304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:33.074390 21301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:33.074424 21302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:33.074941 21296 server_base.cc:1034] running on GCE node
I20260927 15:06:33.075660 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:33.076987 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:33.078271 21296 hybrid_clock.cc:648] HybridClock initialized: now 1790521593078248 us; error 40 us; skew 500 ppm
Sep 27 15:06:33 dist-test-slave-ml7t krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:33.082903 21296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:33.084054 21296 webserver.cc:460] Webserver started at http://127.0.0.1:46405/ using document root <none> and password file <none>
I20260927 15:06:33.084743 21296 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:33.084815 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:33.085050 21296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:33.087481 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "aebb3f3ec0de4ac3bb881060b120ad8c"
format_stamp: "Formatted at 2026-09-27 15:06:33 on dist-test-slave-ml7t"
server_key: "d3fea89ddfe1e9694ba7bcdd8879add4"
server_key_iv: "4b9ecd02bf3d809e93a52c527f870e4e"
server_key_version: "encryptionkey@0"
I20260927 15:06:33.088106 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "aebb3f3ec0de4ac3bb881060b120ad8c"
format_stamp: "Formatted at 2026-09-27 15:06:33 on dist-test-slave-ml7t"
server_key: "d3fea89ddfe1e9694ba7bcdd8879add4"
server_key_iv: "4b9ecd02bf3d809e93a52c527f870e4e"
server_key_version: "encryptionkey@0"
I20260927 15:06:33.092233 21296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:06:33.094954 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:33.096637 21296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:33.096803 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "aebb3f3ec0de4ac3bb881060b120ad8c"
format_stamp: "Formatted at 2026-09-27 15:06:33 on dist-test-slave-ml7t"
server_key: "d3fea89ddfe1e9694ba7bcdd8879add4"
server_key_iv: "4b9ecd02bf3d809e93a52c527f870e4e"
server_key_version: "encryptionkey@0"
I20260927 15:06:33.096946 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:33.115965 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:33.119277 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:33.119431 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:33.126953 21296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38919
I20260927 15:06:33.126969 21363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38919 every 8 connection(s)
I20260927 15:06:33.128021 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:33.131173 21364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:33.137034 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c: Bootstrap starting.
I20260927 15:06:33.138289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21296
I20260927 15:06:33.138449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:33.138743 392 external_mini_cluster.cc:1428] Setting key f9d482b7f5cbc343618d96f7a25387fe
I20260927 15:06:33.139895 21364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:33.140818 21364 log.cc:824] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:33.143041 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c: No bootstrap required, opened a new log
I20260927 15:06:33.147441 21364 raft_consensus.cc:348] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38919 } }
I20260927 15:06:33.147694 21364 raft_consensus.cc:374] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:33.147787 21364 raft_consensus.cc:729] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebb3f3ec0de4ac3bb881060b120ad8c, State: Initialized, Role: FOLLOWER
I20260927 15:06:33.148335 21364 consensus_queue.cc:260] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38919 } }
I20260927 15:06:33.148509 21364 raft_consensus.cc:388] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 27 15:06:33 dist-test-slave-ml7t krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:06:33.148576 21364 raft_consensus.cc:482] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:33.148676 21364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:33.149883 21364 raft_consensus.cc:504] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38919 } }
I20260927 15:06:33.150336 21364 leader_election.cc:302] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aebb3f3ec0de4ac3bb881060b120ad8c; no voters:
I20260927 15:06:33.150744 21364 leader_election.cc:288] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:33.150816 21369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:33.151024 21369 raft_consensus.cc:686] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [term 1 LEADER]: Becoming Leader. State: Replica: aebb3f3ec0de4ac3bb881060b120ad8c, State: Running, Role: LEADER
I20260927 15:06:33.151397 21369 consensus_queue.cc:237] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38919 } }
I20260927 15:06:33.152100 21364 sys_catalog.cc:564] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:33.153083 21370 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aebb3f3ec0de4ac3bb881060b120ad8c. Latest consensus state: current_term: 1 leader_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38919 } } }
I20260927 15:06:33.153299 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebb3f3ec0de4ac3bb881060b120ad8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38919 } } }
I20260927 15:06:33.153539 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:33.153440 21370 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:33.155310 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:33.140003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:53431 (local address 127.0.0.1:38919)
0927 15:06:33.140526 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:33.140545 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:33.140592 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:06:33.141409 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:33.141428 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:33.141883 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:33.142255 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:33.142940 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.144398 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.145301 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.146174 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.146366 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:33.149323 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:33.149328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:33.149341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:33.149366 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:33.151600 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.152225 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.152231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.152237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.152359 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.152765 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.152771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.152774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.153376 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:33.153650 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:33.154100 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 15:06:33.154414 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":204,"threads_started":1}
I20260927 15:06:33.158793 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 15:06:33.158897 21384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:33.158975 21384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:33.159070 21379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:33.159740 21379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:33.165416 21379 catalog_manager.cc:1347] Generated new cluster ID: 28463b2663dd44ffaecff2f28600e9dc
I20260927 15:06:33.165484 21379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:33.173182 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:33.163997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:32769 (local address 127.0.0.1:38919)
0927 15:06:33.164101 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:33.164106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:33.164318 (+ 212us) server_negotiation.cc:407] Connection header received
0927 15:06:33.164471 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:33.164475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:33.164520 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:33.164614 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:33.165396 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.166189 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.167356 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:33.167747 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:33.167863 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:33.169491 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:33.169497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:33.169501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:33.169526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:33.171377 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.171882 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.171885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.171887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.171982 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:33.172315 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:33.172318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:33.172320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:33.172534 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:33.172685 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:33.172928 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 15:06:33.173023 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 15:06:33.220817 21379 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:33.222572 21379 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:33.231920 21379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aebb3f3ec0de4ac3bb881060b120ad8c: Generated new TSK 0
I20260927 15:06:33.232774 21379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:33.241174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21296
W20260927 15:06:33.246778 21281 connection.cc:535] client connection to 127.0.0.1:38919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38919): Connection reset by peer (error 104)
2026-09-27T15:06:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:33 dist-test-slave-ml7t krb5kdc[21395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:33 dist-test-slave-ml7t krb5kdc[21395](info): set up 2 sockets
Sep 27 15:06:33 dist-test-slave-ml7t krb5kdc[21395](info): commencing operation
krb5kdc: starting...
W20260927 15:06:34.282140 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:34 dist-test-slave-ml7t krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:34.406821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:34.518842 21408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:34.519147 21408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:34.519229 21408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:34.523188 21408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:34.523281 21408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:34.523320 21408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:34.523378 21408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:34.523418 21408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:34.528546 21408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:34.529601 21408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:34.530654 21408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:34.539060 21414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:34.539074 21416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:34.539271 21408 server_base.cc:1034] running on GCE node
W20260927 15:06:34.539078 21413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:34.539928 21408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:34.541077 21408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:34.542259 21408 hybrid_clock.cc:648] HybridClock initialized: now 1790521594542231 us; error 40 us; skew 500 ppm
Sep 27 15:06:34 dist-test-slave-ml7t krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:34.545094 21408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:34.546056 21408 webserver.cc:460] Webserver started at http://127.0.0.1:46875/ using document root <none> and password file <none>
I20260927 15:06:34.546605 21408 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:34.546694 21408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:34.546922 21408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:34.548902 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "530a24fcb3b5422d8e412d6e63152c18"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-ml7t"
server_key: "0a0aae01473fe7559c78f26bbad21e01"
server_key_iv: "d5f597f3dc2a804b1e9f0dea89d5a7fd"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.549467 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "530a24fcb3b5422d8e412d6e63152c18"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-ml7t"
server_key: "0a0aae01473fe7559c78f26bbad21e01"
server_key_iv: "d5f597f3dc2a804b1e9f0dea89d5a7fd"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.553331 21408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:06:34.555914 21423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:34.557269 21408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:06:34.557425 21408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "530a24fcb3b5422d8e412d6e63152c18"
format_stamp: "Formatted at 2026-09-27 15:06:34 on dist-test-slave-ml7t"
server_key: "0a0aae01473fe7559c78f26bbad21e01"
server_key_iv: "d5f597f3dc2a804b1e9f0dea89d5a7fd"
server_key_version: "encryptionkey@0"
I20260927 15:06:34.557548 21408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:34.572516 21408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:34.575892 21408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:34.576104 21408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:34.583336 21408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38109
I20260927 15:06:34.583377 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38109 every 8 connection(s)
I20260927 15:06:34.584287 21408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:34.587059 21476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:34.592566 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18: Bootstrap starting.
I20260927 15:06:34.593344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21408
I20260927 15:06:34.593513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:34.593899 392 external_mini_cluster.cc:1428] Setting key 2020842b6d15cd7fb652d84190f8342b
I20260927 15:06:34.595598 21476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:34.596354 21476 log.cc:824] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:34.598407 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18: No bootstrap required, opened a new log
I20260927 15:06:34.602632 21476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530a24fcb3b5422d8e412d6e63152c18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260927 15:06:34.602773 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.595092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:41005 (local address 127.0.0.1:38109)
0927 15:06:34.595525 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.595539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.595573 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:06:34.596274 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.596290 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.596672 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.597056 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:34.597396 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.598402 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.599612 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.600096 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.600279 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.601147 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:34.601152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:34.601162 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:34.601528 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:34.601575 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:34.601596 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:34.601694 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.601895 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.602061 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260927 15:06:34.602859 21476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:34.602928 21476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530a24fcb3b5422d8e412d6e63152c18, State: Initialized, Role: FOLLOWER
I20260927 15:06:34.603328 21476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530a24fcb3b5422d8e412d6e63152c18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260927 15:06:34.603439 21476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:34.603519 21476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:34.603612 21476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:34.604638 21476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530a24fcb3b5422d8e412d6e63152c18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260927 15:06:34.604959 21476 leader_election.cc:302] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 530a24fcb3b5422d8e412d6e63152c18; no voters:
I20260927 15:06:34.605278 21476 leader_election.cc:288] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:34.605378 21481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:34.605580 21481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [term 1 LEADER]: Becoming Leader. State: Replica: 530a24fcb3b5422d8e412d6e63152c18, State: Running, Role: LEADER
I20260927 15:06:34.605719 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:34.605904 21481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530a24fcb3b5422d8e412d6e63152c18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260927 15:06:34.606518 21476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:34.606954 21481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 530a24fcb3b5422d8e412d6e63152c18. Latest consensus state: current_term: 1 leader_uuid: "530a24fcb3b5422d8e412d6e63152c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530a24fcb3b5422d8e412d6e63152c18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } } }
I20260927 15:06:34.607069 21481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:34.607373 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "530a24fcb3b5422d8e412d6e63152c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530a24fcb3b5422d8e412d6e63152c18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } } }
I20260927 15:06:34.607447 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:34.607460 21489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:34.610430 21489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:34.612838 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:34.607996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:52915 (local address 127.0.0.1:38109)
0927 15:06:34.608114 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:34.608118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:34.608366 (+ 248us) server_negotiation.cc:407] Connection header received
0927 15:06:34.608547 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:34.608551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:34.608598 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:34.608653 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:34.609100 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.609820 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.611041 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:34.611357 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:34.611455 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:34.612365 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:34.612369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:34.612371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:34.612384 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:34.612410 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:34.612425 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:34.612513 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:34.612687 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:06:34.612720 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:06:34.615922 21489 catalog_manager.cc:1347] Generated new cluster ID: 97ec12e8c13f46b7978cbf7eaf272353
I20260927 15:06:34.615985 21489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:34.621731 21489 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:34.622869 21489 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:34.633210 21489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 530a24fcb3b5422d8e412d6e63152c18: Generated new TSK 0
I20260927 15:06:34.633973 21489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:34.642294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21408
W20260927 15:06:34.646945 21393 connection.cc:535] client connection to 127.0.0.1:38109 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38109): Connection reset by peer (error 104)
2026-09-27T15:06:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:34 dist-test-slave-ml7t krb5kdc[21507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:34 dist-test-slave-ml7t krb5kdc[21507](info): set up 2 sockets
Sep 27 15:06:34 dist-test-slave-ml7t krb5kdc[21507](info): commencing operation
krb5kdc: starting...
W20260927 15:06:35.674068 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:35 dist-test-slave-ml7t krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:35.796468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:35.905877 21520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:35.906116 21520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:35.906155 21520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:35.910094 21520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:35.910147 21520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:35.910166 21520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:35.910182 21520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:35.910197 21520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:35.915355 21520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:35.916275 21520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:35.917181 21520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:35.925618 21525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:35.925721 21526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:35.925874 21520 server_base.cc:1034] running on GCE node
W20260927 15:06:35.925628 21528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:35.926507 21520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:35.927541 21520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:35.928726 21520 hybrid_clock.cc:648] HybridClock initialized: now 1790521595928701 us; error 48 us; skew 500 ppm
Sep 27 15:06:35 dist-test-slave-ml7t krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:35.931581 21520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:35.932487 21520 webserver.cc:460] Webserver started at http://127.0.0.1:35125/ using document root <none> and password file <none>
I20260927 15:06:35.933012 21520 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:35.933073 21520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:35.933235 21520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:35.935292 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "91687d76cb7b4336a3295758fb7e0142"
format_stamp: "Formatted at 2026-09-27 15:06:35 on dist-test-slave-ml7t"
server_key: "10e9d41ca3645a97b6d73d2cbbd1f0da"
server_key_iv: "1707cca852db18377d6f51664b4a3b65"
server_key_version: "encryptionkey@0"
I20260927 15:06:35.935807 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "91687d76cb7b4336a3295758fb7e0142"
format_stamp: "Formatted at 2026-09-27 15:06:35 on dist-test-slave-ml7t"
server_key: "10e9d41ca3645a97b6d73d2cbbd1f0da"
server_key_iv: "1707cca852db18377d6f51664b4a3b65"
server_key_version: "encryptionkey@0"
I20260927 15:06:35.939555 21520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:06:35.942130 21535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:35.943437 21520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:06:35.943535 21520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "91687d76cb7b4336a3295758fb7e0142"
format_stamp: "Formatted at 2026-09-27 15:06:35 on dist-test-slave-ml7t"
server_key: "10e9d41ca3645a97b6d73d2cbbd1f0da"
server_key_iv: "1707cca852db18377d6f51664b4a3b65"
server_key_version: "encryptionkey@0"
I20260927 15:06:35.943621 21520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:35.948081 21520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:35.951113 21520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:35.951258 21520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:35.958541 21520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40607
I20260927 15:06:35.958570 21587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40607 every 8 connection(s)
I20260927 15:06:35.959511 21520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:35.962505 21588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:35.962602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21520
I20260927 15:06:35.962776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:35.963064 392 external_mini_cluster.cc:1428] Setting key 3ac3fe36894e70bd9cfd170691fbdaf0
I20260927 15:06:35.968446 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.964463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:54209 (local address 127.0.0.1:40607)
0927 15:06:35.964893 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.964906 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.964939 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:06:35.965868 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.965887 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.966163 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.966518 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:35.967338 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:35.967343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:35.967354 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:35.967364 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:35.967414 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:35.967432 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:35.967576 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.967645 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.967804 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260927 15:06:35.969604 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142: Bootstrap starting.
I20260927 15:06:35.971398 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:35.972178 21588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:35.973006 21588 log.cc:824] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:35.975319 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:35.973546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:60821 (local address 127.0.0.1:40607)
0927 15:06:35.973674 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:35.973678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:35.973980 (+ 302us) server_negotiation.cc:407] Connection header received
0927 15:06:35.974116 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:35.974119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:35.974146 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:35.974210 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:35.974917 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:35.974920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:35.974922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:35.974932 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:35.974955 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:35.974964 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:35.975031 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:35.975142 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 15:06:35.975173 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:06:35.975519 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142: No bootstrap required, opened a new log
I20260927 15:06:35.979379 21588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91687d76cb7b4336a3295758fb7e0142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 15:06:35.979645 21588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:35.979709 21588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91687d76cb7b4336a3295758fb7e0142, State: Initialized, Role: FOLLOWER
I20260927 15:06:35.980149 21588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91687d76cb7b4336a3295758fb7e0142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 15:06:35.980317 21588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:35.980409 21588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:35.980515 21588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:35.981626 21588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91687d76cb7b4336a3295758fb7e0142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 15:06:35.982017 21588 leader_election.cc:302] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91687d76cb7b4336a3295758fb7e0142; no voters:
I20260927 15:06:35.982353 21588 leader_election.cc:288] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:35.982530 21598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:35.982764 21598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [term 1 LEADER]: Becoming Leader. State: Replica: 91687d76cb7b4336a3295758fb7e0142, State: Running, Role: LEADER
I20260927 15:06:35.983165 21598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91687d76cb7b4336a3295758fb7e0142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 15:06:35.983670 21588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:35.984929 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91687d76cb7b4336a3295758fb7e0142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91687d76cb7b4336a3295758fb7e0142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } } }
I20260927 15:06:35.985077 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:35.985400 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91687d76cb7b4336a3295758fb7e0142. Latest consensus state: current_term: 1 leader_uuid: "91687d76cb7b4336a3295758fb7e0142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91687d76cb7b4336a3295758fb7e0142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } } }
I20260927 15:06:35.985504 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:35.985484 21607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:35.989105 21607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:35.995095 21607 catalog_manager.cc:1347] Generated new cluster ID: d4226cc031b74c1f960d80c3747118ac
I20260927 15:06:35.995173 21607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:36.020095 21607 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:36.020521 21607 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:36.036625 21607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91687d76cb7b4336a3295758fb7e0142: Generated new TSK 0
I20260927 15:06:36.037534 21607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:36.044881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21520
2026-09-27T15:06:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:36 dist-test-slave-ml7t krb5kdc[21619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:36 dist-test-slave-ml7t krb5kdc[21619](info): set up 2 sockets
Sep 27 15:06:36 dist-test-slave-ml7t krb5kdc[21619](info): commencing operation
krb5kdc: starting...
W20260927 15:06:37.078092 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:37 dist-test-slave-ml7t krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:37.199069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:37.307246 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:37.307512 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:37.307554 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:37.311575 21632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:37.311633 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:37.311653 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:37.311669 21632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:37.311684 21632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:37.316886 21632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:37.318017 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:37.319043 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:37.327699 21638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:37.327797 21640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:37.327767 21632 server_base.cc:1034] running on GCE node
W20260927 15:06:37.327747 21637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:37.328477 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:37.329617 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:37.330808 21632 hybrid_clock.cc:648] HybridClock initialized: now 1790521597330801 us; error 31 us; skew 500 ppm
Sep 27 15:06:37 dist-test-slave-ml7t krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:37.333705 21632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:37.334625 21632 webserver.cc:460] Webserver started at http://127.0.0.1:40407/ using document root <none> and password file <none>
I20260927 15:06:37.335166 21632 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:37.335299 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:37.335521 21632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:37.337507 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "c23140a1390945d284476d52ec9f7264"
format_stamp: "Formatted at 2026-09-27 15:06:37 on dist-test-slave-ml7t"
server_key: "36a0c46fedc91ec1a3e04dcc032d7eaa"
server_key_iv: "5cdc6eaea9466647623b25a11904bd36"
server_key_version: "encryptionkey@0"
I20260927 15:06:37.338069 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "c23140a1390945d284476d52ec9f7264"
format_stamp: "Formatted at 2026-09-27 15:06:37 on dist-test-slave-ml7t"
server_key: "36a0c46fedc91ec1a3e04dcc032d7eaa"
server_key_iv: "5cdc6eaea9466647623b25a11904bd36"
server_key_version: "encryptionkey@0"
I20260927 15:06:37.341668 21632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:37.343948 21647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:37.345108 21632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:37.345279 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "c23140a1390945d284476d52ec9f7264"
format_stamp: "Formatted at 2026-09-27 15:06:37 on dist-test-slave-ml7t"
server_key: "36a0c46fedc91ec1a3e04dcc032d7eaa"
server_key_iv: "5cdc6eaea9466647623b25a11904bd36"
server_key_version: "encryptionkey@0"
I20260927 15:06:37.345391 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:37.369882 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:37.373292 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:37.373564 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:37.381928 21632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43033
I20260927 15:06:37.381963 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43033 every 8 connection(s)
I20260927 15:06:37.382982 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:37.384902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21632
I20260927 15:06:37.385165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:37.385489 392 external_mini_cluster.cc:1428] Setting key 1c8aee45c7e334eb89ca67e629075480
I20260927 15:06:37.386029 21700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:37.391992 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264: Bootstrap starting.
Sep 27 15:06:37 dist-test-slave-ml7t krb5kdc[21619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:06:37.394480 21700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:37.395364 21700 log.cc:824] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:37.397850 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264: No bootstrap required, opened a new log
I20260927 15:06:37.400131 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:37.386987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:33765 (local address 127.0.0.1:43033)
0927 15:06:37.387450 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:37.387464 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:37.387500 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:06:37.388147 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:37.388163 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:37.388508 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:37.388864 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:37.389656 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.390693 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.391565 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.392068 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.392241 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:37.395087 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:37.395092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:37.395105 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:37.395131 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:37.396870 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:37.397516 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:37.397520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:37.397524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:37.397630 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:37.398058 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:37.398061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:37.398063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:37.398474 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:37.398642 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:37.399089 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 15:06:37.399342 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":161,"threads_started":1}
I20260927 15:06:37.402438 21700 raft_consensus.cc:348] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23140a1390945d284476d52ec9f7264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43033 } }
I20260927 15:06:37.402765 21700 raft_consensus.cc:374] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:37.402840 21700 raft_consensus.cc:729] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23140a1390945d284476d52ec9f7264, State: Initialized, Role: FOLLOWER
I20260927 15:06:37.403455 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:37.403373 21700 consensus_queue.cc:260] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23140a1390945d284476d52ec9f7264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43033 } }
I20260927 15:06:37.403558 21700 raft_consensus.cc:388] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:37.403625 21700 raft_consensus.cc:482] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:37.403726 21700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:37.405100 21700 raft_consensus.cc:504] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23140a1390945d284476d52ec9f7264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43033 } }
I20260927 15:06:37.405491 21700 leader_election.cc:302] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c23140a1390945d284476d52ec9f7264; no voters:
I20260927 15:06:37.405902 21700 leader_election.cc:288] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:37.405982 21709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:37.406291 21709 raft_consensus.cc:686] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [term 1 LEADER]: Becoming Leader. State: Replica: c23140a1390945d284476d52ec9f7264, State: Running, Role: LEADER
I20260927 15:06:37.406668 21709 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23140a1390945d284476d52ec9f7264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43033 } }
I20260927 15:06:37.407393 21700 sys_catalog.cc:564] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:37.408036 21709 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c23140a1390945d284476d52ec9f7264. Latest consensus state: current_term: 1 leader_uuid: "c23140a1390945d284476d52ec9f7264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23140a1390945d284476d52ec9f7264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43033 } } }
I20260927 15:06:37.408150 21709 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:37.408641 21711 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c23140a1390945d284476d52ec9f7264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23140a1390945d284476d52ec9f7264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43033 } } }
I20260927 15:06:37.408839 21711 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:37.409175 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:37.412724 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:37.415510 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:37.405901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:58859 (local address 127.0.0.1:43033)
0927 15:06:37.406031 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:37.406036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:37.406319 (+ 283us) server_negotiation.cc:407] Connection header received
0927 15:06:37.406596 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:37.406599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:37.406647 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:37.406699 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:37.407636 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.408412 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.409552 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:37.409931 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:37.410006 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:37.411807 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:37.411811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:37.411813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:37.411838 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:37.413619 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:37.414203 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:37.414206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:37.414208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:37.414302 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:37.414630 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:37.414633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:37.414635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:37.414812 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:37.414964 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:37.415232 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 15:06:37.415333 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:37.418929 21715 catalog_manager.cc:1347] Generated new cluster ID: b88479a480f14b3b847a49b85cc6dfe3
I20260927 15:06:37.418989 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:37.427451 21715 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:37.428745 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:37.434188 21715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c23140a1390945d284476d52ec9f7264: Generated new TSK 0
I20260927 15:06:37.434836 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:37.443262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21632
W20260927 15:06:37.448601 21617 connection.cc:535] client connection to 127.0.0.1:43033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43033): Connection reset by peer (error 104)
2026-09-27T15:06:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:37 dist-test-slave-ml7t krb5kdc[21731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:37 dist-test-slave-ml7t krb5kdc[21731](info): set up 2 sockets
Sep 27 15:06:37 dist-test-slave-ml7t krb5kdc[21731](info): commencing operation
krb5kdc: starting...
W20260927 15:06:38.573989 392 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:38 dist-test-slave-ml7t krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:38.696875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:38.805275 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:38.805616 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:38.805704 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:38.809800 21744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:38.809952 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:38.810002 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:38.810031 21744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:38.810078 21744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:38.815455 21744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:38.816516 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:38.817533 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:38.826059 21750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:38.826066 21752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:38.826164 21744 server_base.cc:1034] running on GCE node
W20260927 15:06:38.826066 21749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:38.827025 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:38.828081 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:38.829291 21744 hybrid_clock.cc:648] HybridClock initialized: now 1790521598829271 us; error 31 us; skew 500 ppm
Sep 27 15:06:38 dist-test-slave-ml7t krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:38.836145 21744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:38.837097 21744 webserver.cc:460] Webserver started at http://127.0.0.1:46435/ using document root <none> and password file <none>
I20260927 15:06:38.837659 21744 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:38.837754 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:38.837973 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:38.840057 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "0981e06844644e408b7429555679bf19"
format_stamp: "Formatted at 2026-09-27 15:06:38 on dist-test-slave-ml7t"
server_key: "7f2ad44608e2c8a58a17d4be4395f443"
server_key_iv: "18e2533c66f5ad0c9ded2a8e574bedb5"
server_key_version: "encryptionkey@0"
I20260927 15:06:38.840631 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "0981e06844644e408b7429555679bf19"
format_stamp: "Formatted at 2026-09-27 15:06:38 on dist-test-slave-ml7t"
server_key: "7f2ad44608e2c8a58a17d4be4395f443"
server_key_iv: "18e2533c66f5ad0c9ded2a8e574bedb5"
server_key_version: "encryptionkey@0"
I20260927 15:06:38.844563 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:06:38.847011 21759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:38.848165 21744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:38.848351 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "0981e06844644e408b7429555679bf19"
format_stamp: "Formatted at 2026-09-27 15:06:38 on dist-test-slave-ml7t"
server_key: "7f2ad44608e2c8a58a17d4be4395f443"
server_key_iv: "18e2533c66f5ad0c9ded2a8e574bedb5"
server_key_version: "encryptionkey@0"
I20260927 15:06:38.848464 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:38.867357 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:38.870960 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:38.871171 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:38.879019 21744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35941
I20260927 15:06:38.879038 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35941 every 8 connection(s)
I20260927 15:06:38.880034 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:38.882794 21812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:38.883068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21744
I20260927 15:06:38.883211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:38.883509 392 external_mini_cluster.cc:1428] Setting key 5500fe6c22c8e28fa03dfe9469bfde69
I20260927 15:06:38.889073 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19: Bootstrap starting.
I20260927 15:06:38.891273 21812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:38.892105 21812 log.cc:824] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:38.892495 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:38.884888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:35583 (local address 127.0.0.1:35941)
0927 15:06:38.885299 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:38.885313 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:38.885348 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:06:38.886097 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:38.886116 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:38.886458 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:38.886822 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:38.887211 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:38.888176 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:38.889094 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:38.889660 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:38.889877 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:38.890777 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:38.890781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:38.890793 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:38.891175 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:38.891230 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:38.891253 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:38.891390 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:38.891579 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 15:06:38.891719 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
I20260927 15:06:38.894019 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19: No bootstrap required, opened a new log
I20260927 15:06:38.895455 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:38.898308 21812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0981e06844644e408b7429555679bf19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 15:06:38.898631 21812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:38.898716 21812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0981e06844644e408b7429555679bf19, State: Initialized, Role: FOLLOWER
I20260927 15:06:38.899150 21812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0981e06844644e408b7429555679bf19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 15:06:38.899286 21812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:38.899371 21812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:38.899494 21812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:38.900441 21812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0981e06844644e408b7429555679bf19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 15:06:38.900817 21812 leader_election.cc:302] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0981e06844644e408b7429555679bf19; no voters:
I20260927 15:06:38.901188 21812 leader_election.cc:288] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:38.901407 21822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:38.901644 21822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [term 1 LEADER]: Becoming Leader. State: Replica: 0981e06844644e408b7429555679bf19, State: Running, Role: LEADER
I20260927 15:06:38.902024 21822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0981e06844644e408b7429555679bf19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 15:06:38.902163 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:38.897659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:44993 (local address 127.0.0.1:35941)
0927 15:06:38.897842 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:38.897845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:38.898037 (+ 192us) server_negotiation.cc:407] Connection header received
0927 15:06:38.898163 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:38.898167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:38.898211 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:38.898294 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:38.898652 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:38.899391 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:38.900276 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:38.900529 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:38.900653 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:38.901544 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:38.901548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:38.901551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:38.901563 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:38.901589 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:38.901601 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:38.901657 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:38.901920 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 15:06:38.901957 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:06:38.902683 21812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:38.903414 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0981e06844644e408b7429555679bf19. Latest consensus state: current_term: 1 leader_uuid: "0981e06844644e408b7429555679bf19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0981e06844644e408b7429555679bf19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } } }
I20260927 15:06:38.903604 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:38.903867 21824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0981e06844644e408b7429555679bf19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0981e06844644e408b7429555679bf19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } } }
I20260927 15:06:38.903965 21824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:38.904058 21831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:38.907248 21831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:38.913076 21831 catalog_manager.cc:1347] Generated new cluster ID: 5201677e07ed43f5b4a642092d862638
I20260927 15:06:38.913146 21831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:38.959419 21831 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:38.960759 21831 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:38.970544 21831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0981e06844644e408b7429555679bf19: Generated new TSK 0
I20260927 15:06:38.971318 21831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:38.982312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21744
W20260927 15:06:38.989395 21729 connection.cc:535] client connection to 127.0.0.1:35941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35941): Connection reset by peer (error 104)
2026-09-27T15:06:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:39 dist-test-slave-ml7t krb5kdc[21843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:39 dist-test-slave-ml7t krb5kdc[21843](info): set up 2 sockets
Sep 27 15:06:39 dist-test-slave-ml7t krb5kdc[21843](info): commencing operation
krb5kdc: starting...
W20260927 15:06:40.022447 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:40 dist-test-slave-ml7t krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:06:40.147189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:40.258229 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:40.258630 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:40.258701 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:40.262714 21856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:40.262804 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:40.262842 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:40.262879 21856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:40.262917 21856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:40.268299 21856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:40.269402 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:40.270534 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:40.279065 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:40.279075 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:40.279134 21856 server_base.cc:1034] running on GCE node
W20260927 15:06:40.279076 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:40.279933 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:40.281064 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:40.282255 21856 hybrid_clock.cc:648] HybridClock initialized: now 1790521600282231 us; error 41 us; skew 500 ppm
Sep 27 15:06:40 dist-test-slave-ml7t krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:40.284955 21856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:06:40.285809 21856 webserver.cc:460] Webserver started at http://127.0.0.1:45057/ using document root <none> and password file <none>
I20260927 15:06:40.286338 21856 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:40.286396 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:40.286551 21856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:40.288478 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "f522191dcb49472985c2f65e1e91117d"
format_stamp: "Formatted at 2026-09-27 15:06:40 on dist-test-slave-ml7t"
server_key: "a3a812e2953a8219dcf71d4fa0825471"
server_key_iv: "142a40c6d40b5d5d08747b1a7f12fef0"
server_key_version: "encryptionkey@0"
I20260927 15:06:40.288996 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "f522191dcb49472985c2f65e1e91117d"
format_stamp: "Formatted at 2026-09-27 15:06:40 on dist-test-slave-ml7t"
server_key: "a3a812e2953a8219dcf71d4fa0825471"
server_key_iv: "142a40c6d40b5d5d08747b1a7f12fef0"
server_key_version: "encryptionkey@0"
I20260927 15:06:40.292534 21856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:06:40.295004 21871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:40.296253 21856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:40.296409 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "f522191dcb49472985c2f65e1e91117d"
format_stamp: "Formatted at 2026-09-27 15:06:40 on dist-test-slave-ml7t"
server_key: "a3a812e2953a8219dcf71d4fa0825471"
server_key_iv: "142a40c6d40b5d5d08747b1a7f12fef0"
server_key_version: "encryptionkey@0"
I20260927 15:06:40.296506 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:40.300969 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:40.303987 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:40.304172 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:40.310786 21856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37983
I20260927 15:06:40.310822 21923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37983 every 8 connection(s)
I20260927 15:06:40.311667 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:40.313081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21856
I20260927 15:06:40.313256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:40.313585 392 external_mini_cluster.cc:1428] Setting key 898238c8bf10a833f6dd37658aa87e5b
I20260927 15:06:40.314548 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:40.318751 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:40.315017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:45329 (local address 127.0.0.1:37983)
0927 15:06:40.315457 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:40.315472 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:40.315510 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:06:40.316187 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:40.316204 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:40.316476 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:40.316834 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:40.317548 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:40.317553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:40.317578 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:40.317594 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:40.317649 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:40.317668 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:40.317846 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:40.317920 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:06:40.318078 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260927 15:06:40.320403 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d: Bootstrap starting.
I20260927 15:06:40.321460 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:40.322775 21924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:40.323541 21924 log.cc:824] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:40.325356 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:40.323701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:33821 (local address 127.0.0.1:37983)
0927 15:06:40.323844 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:40.323849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:40.323997 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:06:40.324133 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:40.324136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:40.324173 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:40.324267 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:40.324982 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:40.324985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:40.324988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:40.324999 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:40.325029 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:40.325039 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:40.325127 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:40.325200 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:06:40.325240 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:06:40.325709 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d: No bootstrap required, opened a new log
I20260927 15:06:40.329203 21924 raft_consensus.cc:348] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522191dcb49472985c2f65e1e91117d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260927 15:06:40.329419 21924 raft_consensus.cc:374] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:40.329496 21924 raft_consensus.cc:729] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f522191dcb49472985c2f65e1e91117d, State: Initialized, Role: FOLLOWER
I20260927 15:06:40.329963 21924 consensus_queue.cc:260] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522191dcb49472985c2f65e1e91117d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260927 15:06:40.330104 21924 raft_consensus.cc:388] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:40.330194 21924 raft_consensus.cc:482] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:40.330308 21924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:40.331355 21924 raft_consensus.cc:504] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522191dcb49472985c2f65e1e91117d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260927 15:06:40.331679 21924 leader_election.cc:302] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f522191dcb49472985c2f65e1e91117d; no voters:
I20260927 15:06:40.332010 21924 leader_election.cc:288] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:40.332127 21934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:40.332389 21934 raft_consensus.cc:686] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [term 1 LEADER]: Becoming Leader. State: Replica: f522191dcb49472985c2f65e1e91117d, State: Running, Role: LEADER
I20260927 15:06:40.332726 21934 consensus_queue.cc:237] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522191dcb49472985c2f65e1e91117d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } }
I20260927 15:06:40.333174 21924 sys_catalog.cc:564] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:40.334270 21935 sys_catalog.cc:455] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f522191dcb49472985c2f65e1e91117d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522191dcb49472985c2f65e1e91117d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } } }
I20260927 15:06:40.334394 21935 sys_catalog.cc:458] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:40.334690 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f522191dcb49472985c2f65e1e91117d. Latest consensus state: current_term: 1 leader_uuid: "f522191dcb49472985c2f65e1e91117d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522191dcb49472985c2f65e1e91117d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37983 } } }
I20260927 15:06:40.334764 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:40.334787 21943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:40.337924 21943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:40.343930 21943 catalog_manager.cc:1347] Generated new cluster ID: 6ae8f27897404b1ea9246797a340f094
I20260927 15:06:40.343993 21943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:40.351905 21943 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:40.352276 21943 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:40.357688 21943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f522191dcb49472985c2f65e1e91117d: Generated new TSK 0
I20260927 15:06:40.358588 21943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:40.366945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21856
2026-09-27T15:06:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:40 dist-test-slave-ml7t krb5kdc[21955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:40 dist-test-slave-ml7t krb5kdc[21955](info): set up 2 sockets
Sep 27 15:06:40 dist-test-slave-ml7t krb5kdc[21955](info): commencing operation
krb5kdc: starting...
W20260927 15:06:41.434198 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:41 dist-test-slave-ml7t krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:41.557085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:41.674469 21968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:41.674819 21968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:41.674912 21968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:41.679528 21968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:41.679641 21968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:41.679677 21968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:41.679718 21968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:41.679754 21968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:41.685233 21968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:41.686354 21968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:41.687383 21968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:41.695941 21976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:41.695916 21974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:41.695920 21973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:41.696064 21968 server_base.cc:1034] running on GCE node
I20260927 15:06:41.696862 21968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:41.698040 21968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:41.699246 21968 hybrid_clock.cc:648] HybridClock initialized: now 1790521601699211 us; error 44 us; skew 500 ppm
Sep 27 15:06:41 dist-test-slave-ml7t krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:41.702080 21968 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:41.703054 21968 webserver.cc:460] Webserver started at http://127.0.98.62:44475/ using document root <none> and password file <none>
I20260927 15:06:41.703655 21968 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:41.703755 21968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:41.703994 21968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:41.706151 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "a112ae4089514a54bfeb5d666e6cf1f0"
format_stamp: "Formatted at 2026-09-27 15:06:41 on dist-test-slave-ml7t"
server_key: "f3354504b15ed64533c0bf9628a29012"
server_key_iv: "b20758eaba05d24034a057848e6471d0"
server_key_version: "encryptionkey@0"
I20260927 15:06:41.706728 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "a112ae4089514a54bfeb5d666e6cf1f0"
format_stamp: "Formatted at 2026-09-27 15:06:41 on dist-test-slave-ml7t"
server_key: "f3354504b15ed64533c0bf9628a29012"
server_key_iv: "b20758eaba05d24034a057848e6471d0"
server_key_version: "encryptionkey@0"
I20260927 15:06:41.710464 21968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:06:41.712936 21983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:41.714339 21968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:41.714484 21968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "a112ae4089514a54bfeb5d666e6cf1f0"
format_stamp: "Formatted at 2026-09-27 15:06:41 on dist-test-slave-ml7t"
server_key: "f3354504b15ed64533c0bf9628a29012"
server_key_iv: "b20758eaba05d24034a057848e6471d0"
server_key_version: "encryptionkey@0"
I20260927 15:06:41.714597 21968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:41.729916 21968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:41.732823 21968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:41.733014 21968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:41.739950 21968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33517
I20260927 15:06:41.739964 22035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33517 every 8 connection(s)
I20260927 15:06:41.740850 21968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:41.743551 22036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:41.743870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 21968
I20260927 15:06:41.744014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:41.744298 392 external_mini_cluster.cc:1428] Setting key d91f6f2e9b74fc6f19ea95bc0288ba38
I20260927 15:06:41.750542 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0: Bootstrap starting.
I20260927 15:06:41.752871 22036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0: Neither blocks nor log segments found. Creating new log.
Sep 27 15:06:41 dist-test-slave-ml7t krb5kdc[21955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:41.753746 22036 log.cc:824] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:41.755621 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0: No bootstrap required, opened a new log
I20260927 15:06:41.758703 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:41.745525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44465 (local address 127.0.98.62:33517)
0927 15:06:41.745937 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:41.745956 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:41.745998 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:06:41.746952 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:41.746974 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:41.747392 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:41.747751 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:41.748392 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:41.749905 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:41.750763 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:41.751397 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:41.751557 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:41.754259 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:41.754263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:41.754282 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:41.754307 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:41.756005 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:41.756509 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:41.756514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:41.756518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:41.756618 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:41.756951 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:41.756954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:41.756956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:41.757315 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:41.757524 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:41.757864 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:06:41.758055 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260927 15:06:41.759210 22036 raft_consensus.cc:348] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260927 15:06:41.759467 22036 raft_consensus.cc:374] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:41.759521 22036 raft_consensus.cc:729] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a112ae4089514a54bfeb5d666e6cf1f0, State: Initialized, Role: FOLLOWER
I20260927 15:06:41.759939 22036 consensus_queue.cc:260] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260927 15:06:41.760098 22036 raft_consensus.cc:388] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:41.760183 22036 raft_consensus.cc:482] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:41.760298 22036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:41.761348 22036 raft_consensus.cc:504] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260927 15:06:41.761660 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:41.761699 22036 leader_election.cc:302] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a112ae4089514a54bfeb5d666e6cf1f0; no voters:
I20260927 15:06:41.762106 22036 leader_election.cc:288] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:41.762171 22041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:41.762372 22041 raft_consensus.cc:686] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [term 1 LEADER]: Becoming Leader. State: Replica: a112ae4089514a54bfeb5d666e6cf1f0, State: Running, Role: LEADER
I20260927 15:06:41.762712 22041 consensus_queue.cc:237] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260927 15:06:41.763337 22036 sys_catalog.cc:564] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:41.764302 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } } }
I20260927 15:06:41.764418 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:41.764735 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a112ae4089514a54bfeb5d666e6cf1f0. Latest consensus state: current_term: 1 leader_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a112ae4089514a54bfeb5d666e6cf1f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } } }
I20260927 15:06:41.764812 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:41.765797 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:41.768976 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:41.774168 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:41.764163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42355 (local address 127.0.98.62:33517)
0927 15:06:41.764281 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:41.764286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:41.765479 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 15:06:41.765596 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:41.765600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:41.765645 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:41.765724 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:41.766530 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:41.767256 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:41.768351 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:41.768672 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:41.768777 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:41.770367 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:41.770371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:41.770374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:41.770397 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:41.771852 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:41.772340 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:41.772344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:41.772346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:41.772424 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:41.772821 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:41.772825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:41.772827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:41.773417 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:41.773568 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:41.773914 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:06:41.774015 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:06:41.774703 22055 catalog_manager.cc:1347] Generated new cluster ID: 97c49b60d45b4424a556d326b226af8e
I20260927 15:06:41.774799 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:41.805225 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:41.806265 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:41.817574 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a112ae4089514a54bfeb5d666e6cf1f0: Generated new TSK 0
I20260927 15:06:41.818377 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:41.825963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 21968
W20260927 15:06:41.831148 21953 connection.cc:535] client connection to 127.0.98.62:33517 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33517): Connection reset by peer (error 104)
2026-09-27T15:06:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:41 dist-test-slave-ml7t krb5kdc[22067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:41 dist-test-slave-ml7t krb5kdc[22067](info): set up 2 sockets
Sep 27 15:06:41 dist-test-slave-ml7t krb5kdc[22067](info): commencing operation
krb5kdc: starting...
W20260927 15:06:42.962276 392 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:43 dist-test-slave-ml7t krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:43.083401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35263
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:43.196161 22080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:43.196480 22080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:43.196579 22080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:43.200670 22080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:43.200757 22080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:43.200796 22080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:43.200853 22080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:43.200892 22080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:43.206305 22080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:43.207427 22080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:43.208451 22080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:43.216912 22086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:43.216987 22085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:43.217047 22080 server_base.cc:1034] running on GCE node
W20260927 15:06:43.216986 22088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:43.217813 22080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:43.219009 22080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:43.220297 22080 hybrid_clock.cc:648] HybridClock initialized: now 1790521603220280 us; error 38 us; skew 500 ppm
Sep 27 15:06:43 dist-test-slave-ml7t krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:43.223212 22080 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:43.224084 22080 webserver.cc:460] Webserver started at http://127.0.98.62:41751/ using document root <none> and password file <none>
I20260927 15:06:43.224615 22080 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:43.224674 22080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:43.224853 22080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:43.226974 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "a4865a1b13384007891b0fb424d390b1"
format_stamp: "Formatted at 2026-09-27 15:06:43 on dist-test-slave-ml7t"
server_key: "bb83d1b2735ec08b22c065d00fa6a288"
server_key_iv: "906fa51b8563e4f1680901bd01acbefc"
server_key_version: "encryptionkey@0"
I20260927 15:06:43.227506 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4865a1b13384007891b0fb424d390b1"
format_stamp: "Formatted at 2026-09-27 15:06:43 on dist-test-slave-ml7t"
server_key: "bb83d1b2735ec08b22c065d00fa6a288"
server_key_iv: "906fa51b8563e4f1680901bd01acbefc"
server_key_version: "encryptionkey@0"
I20260927 15:06:43.231282 22080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:06:43.233695 22095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:43.234977 22080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:06:43.235113 22080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "a4865a1b13384007891b0fb424d390b1"
format_stamp: "Formatted at 2026-09-27 15:06:43 on dist-test-slave-ml7t"
server_key: "bb83d1b2735ec08b22c065d00fa6a288"
server_key_iv: "906fa51b8563e4f1680901bd01acbefc"
server_key_version: "encryptionkey@0"
I20260927 15:06:43.235224 22080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:43.247164 22080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:43.250501 22080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:43.250725 22080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:43.257931 22080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35263
I20260927 15:06:43.258075 22147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35263 every 8 connection(s)
I20260927 15:06:43.259196 22080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:43.259913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22080
I20260927 15:06:43.260053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:43.260344 392 external_mini_cluster.cc:1428] Setting key 91a9fb985974eaa108ea4ffa258c88a2
I20260927 15:06:43.262671 22148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:43.268113 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1: Bootstrap starting.
I20260927 15:06:43.269811 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:43.261816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51525 (local address 127.0.98.62:35263)
0927 15:06:43.262262 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:43.262279 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:43.262317 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:06:43.263055 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:43.263074 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:43.263418 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:43.263777 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:43.264191 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.265210 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.266270 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.266757 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.266956 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:43.267983 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:43.267988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:43.268001 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:43.268447 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:43.268505 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:43.268531 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:43.268699 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:43.268895 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 15:06:43.269038 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":142,"threads_started":1}
I20260927 15:06:43.270453 22148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:43.271178 22148 log.cc:824] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:43.272933 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:43.273273 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1: No bootstrap required, opened a new log
I20260927 15:06:43.277806 22148 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4865a1b13384007891b0fb424d390b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260927 15:06:43.278110 22148 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:43.278187 22148 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4865a1b13384007891b0fb424d390b1, State: Initialized, Role: FOLLOWER
I20260927 15:06:43.278707 22148 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4865a1b13384007891b0fb424d390b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260927 15:06:43.278896 22148 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:43.278982 22148 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:43.279098 22148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:43.279661 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:43.275067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33377 (local address 127.0.98.62:35263)
0927 15:06:43.275182 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:43.275185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:43.275478 (+ 293us) server_negotiation.cc:407] Connection header received
0927 15:06:43.275614 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:43.275616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:43.275653 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:43.275732 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:43.276037 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.276525 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.277636 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:43.277954 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:43.278036 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:43.279060 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:43.279064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:43.279066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:43.279080 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:43.279111 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:43.279136 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:43.279230 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:43.279448 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 15:06:43.279507 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:06:43.280301 22148 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4865a1b13384007891b0fb424d390b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260927 15:06:43.280728 22148 leader_election.cc:302] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4865a1b13384007891b0fb424d390b1; no voters:
I20260927 15:06:43.281049 22148 leader_election.cc:288] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:43.281133 22158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:43.281397 22158 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [term 1 LEADER]: Becoming Leader. State: Replica: a4865a1b13384007891b0fb424d390b1, State: Running, Role: LEADER
I20260927 15:06:43.281749 22158 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4865a1b13384007891b0fb424d390b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260927 15:06:43.282560 22148 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:43.283661 22160 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4865a1b13384007891b0fb424d390b1. Latest consensus state: current_term: 1 leader_uuid: "a4865a1b13384007891b0fb424d390b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4865a1b13384007891b0fb424d390b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } } }
I20260927 15:06:43.283654 22159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4865a1b13384007891b0fb424d390b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4865a1b13384007891b0fb424d390b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } } }
I20260927 15:06:43.283797 22160 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:43.283797 22159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:43.284209 22167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:43.286973 22167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:43.293042 22167 catalog_manager.cc:1347] Generated new cluster ID: 719cb4d6d08e48e9a0a4f3c9b8294b72
I20260927 15:06:43.293129 22167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:43.303234 22167 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:43.304217 22167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:43.316148 22167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4865a1b13384007891b0fb424d390b1: Generated new TSK 0
I20260927 15:06:43.316963 22167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:43.324249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22080
W20260927 15:06:43.329506 22065 connection.cc:535] client connection to 127.0.98.62:35263 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35263): Connection reset by peer (error 104)
2026-09-27T15:06:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1498 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:43 dist-test-slave-ml7t krb5kdc[22179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:43 dist-test-slave-ml7t krb5kdc[22179](info): set up 2 sockets
Sep 27 15:06:43 dist-test-slave-ml7t krb5kdc[22179](info): commencing operation
krb5kdc: starting...
W20260927 15:06:44.358126 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:44 dist-test-slave-ml7t krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:44.484790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:44.595400 22192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:44.595670 22192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:44.595714 22192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:44.599756 22192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:44.599828 22192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:44.599848 22192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:44.599865 22192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:44.599880 22192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:44.605283 22192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:44.606420 22192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:44.607445 22192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:44.616093 22198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:44.616179 22197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:44.616441 22192 server_base.cc:1034] running on GCE node
W20260927 15:06:44.616104 22200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:44.616997 22192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:44.618187 22192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:44.619346 22192 hybrid_clock.cc:648] HybridClock initialized: now 1790521604619318 us; error 47 us; skew 500 ppm
Sep 27 15:06:44 dist-test-slave-ml7t krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:44.628401 22192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:44.629438 22192 webserver.cc:460] Webserver started at http://127.0.98.62:36969/ using document root <none> and password file <none>
I20260927 15:06:44.630112 22192 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:44.630187 22192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:44.630426 22192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:44.632663 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "91b937cb309e4134a454007dc336b68f"
format_stamp: "Formatted at 2026-09-27 15:06:44 on dist-test-slave-ml7t"
server_key: "3b446bd74ffa2165a3f4b3a468e4f08c"
server_key_iv: "a7ef80cec7d7d03aa609a6cce8ee40d8"
server_key_version: "encryptionkey@0"
I20260927 15:06:44.633283 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "91b937cb309e4134a454007dc336b68f"
format_stamp: "Formatted at 2026-09-27 15:06:44 on dist-test-slave-ml7t"
server_key: "3b446bd74ffa2165a3f4b3a468e4f08c"
server_key_iv: "a7ef80cec7d7d03aa609a6cce8ee40d8"
server_key_version: "encryptionkey@0"
I20260927 15:06:44.637473 22192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:06:44.640519 22207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:44.641944 22192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:44.642110 22192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "91b937cb309e4134a454007dc336b68f"
format_stamp: "Formatted at 2026-09-27 15:06:44 on dist-test-slave-ml7t"
server_key: "3b446bd74ffa2165a3f4b3a468e4f08c"
server_key_iv: "a7ef80cec7d7d03aa609a6cce8ee40d8"
server_key_version: "encryptionkey@0"
I20260927 15:06:44.642241 22192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:44.647018 22192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:44.650609 22192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:44.650818 22192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:44.658430 22192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41721
I20260927 15:06:44.658437 22259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41721 every 8 connection(s)
I20260927 15:06:44.659467 22192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:44.661060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22192
I20260927 15:06:44.661198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:44.661497 392 external_mini_cluster.cc:1428] Setting key 116e41fd65d00b4f89de998e42cedaa6
I20260927 15:06:44.662654 22260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:44.666873 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:44.662995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40251 (local address 127.0.98.62:41721)
0927 15:06:44.663443 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:44.663458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:44.663495 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:06:44.664128 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:44.664145 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:44.664452 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:44.664808 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:44.665649 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:44.665654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:44.665666 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:44.665676 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:44.665730 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:44.665749 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:44.665926 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:44.666001 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 15:06:44.666167 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260927 15:06:44.668972 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f: Bootstrap starting.
I20260927 15:06:44.670027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:44.671610 22260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:44.672557 22260 log.cc:824] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:44.674091 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:44.672165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59571 (local address 127.0.98.62:41721)
0927 15:06:44.672305 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:44.672308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:44.672538 (+ 230us) server_negotiation.cc:407] Connection header received
0927 15:06:44.672692 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:44.672697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:44.672737 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:44.672811 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:44.673581 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:44.673584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:44.673587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:44.673598 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:44.673625 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:44.673637 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:44.673727 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:44.673937 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 15:06:44.673976 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:06:44.674865 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f: No bootstrap required, opened a new log
I20260927 15:06:44.678644 22260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b937cb309e4134a454007dc336b68f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260927 15:06:44.678893 22260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:44.678996 22260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b937cb309e4134a454007dc336b68f, State: Initialized, Role: FOLLOWER
I20260927 15:06:44.679440 22260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b937cb309e4134a454007dc336b68f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260927 15:06:44.679605 22260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:44.679684 22260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:44.679813 22260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:44.680789 22260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b937cb309e4134a454007dc336b68f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260927 15:06:44.681146 22260 leader_election.cc:302] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91b937cb309e4134a454007dc336b68f; no voters:
I20260927 15:06:44.681502 22260 leader_election.cc:288] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:44.681639 22270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:44.681939 22270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [term 1 LEADER]: Becoming Leader. State: Replica: 91b937cb309e4134a454007dc336b68f, State: Running, Role: LEADER
I20260927 15:06:44.682299 22270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b937cb309e4134a454007dc336b68f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260927 15:06:44.682844 22260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:44.684064 22272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b937cb309e4134a454007dc336b68f. Latest consensus state: current_term: 1 leader_uuid: "91b937cb309e4134a454007dc336b68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b937cb309e4134a454007dc336b68f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } } }
I20260927 15:06:44.684119 22271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91b937cb309e4134a454007dc336b68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b937cb309e4134a454007dc336b68f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } } }
I20260927 15:06:44.684199 22272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:44.684224 22271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:44.684604 22279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:44.687527 22279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:44.693099 22279 catalog_manager.cc:1347] Generated new cluster ID: 210d23ee41524a7fb0d4960498d6df1a
I20260927 15:06:44.693181 22279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:44.709493 22279 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:44.709877 22279 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:44.721223 22279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91b937cb309e4134a454007dc336b68f: Generated new TSK 0
I20260927 15:06:44.722038 22279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:44.727212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22192
2026-09-27T15:06:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:44 dist-test-slave-ml7t krb5kdc[22291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:44 dist-test-slave-ml7t krb5kdc[22291](info): set up 2 sockets
Sep 27 15:06:44 dist-test-slave-ml7t krb5kdc[22291](info): commencing operation
krb5kdc: starting...
W20260927 15:06:45.770180 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:45 dist-test-slave-ml7t krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:45.890643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:45.999809 22304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:46.000084 22304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:46.000126 22304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:46.004048 22304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:46.004118 22304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:46.004137 22304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:46.004154 22304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:46.004169 22304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:46.010054 22304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:46.011237 22304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:46.012317 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:46.021255 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:46.021278 22312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:46.021255 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:46.021551 22304 server_base.cc:1034] running on GCE node
I20260927 15:06:46.022274 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:46.023458 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:46.024622 22304 hybrid_clock.cc:648] HybridClock initialized: now 1790521606024605 us; error 34 us; skew 500 ppm
Sep 27 15:06:46 dist-test-slave-ml7t krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:46.027539 22304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:46.028481 22304 webserver.cc:460] Webserver started at http://127.0.98.62:41627/ using document root <none> and password file <none>
I20260927 15:06:46.029063 22304 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:46.029126 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:46.029381 22304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:46.031490 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "806576f461bd4d77a0261dd872b9c698"
format_stamp: "Formatted at 2026-09-27 15:06:46 on dist-test-slave-ml7t"
server_key: "5b610b172526f76096e5c4eb1e9677d2"
server_key_iv: "4a1edd12ede88b3abe466bed8a96be3d"
server_key_version: "encryptionkey@0"
I20260927 15:06:46.032066 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "806576f461bd4d77a0261dd872b9c698"
format_stamp: "Formatted at 2026-09-27 15:06:46 on dist-test-slave-ml7t"
server_key: "5b610b172526f76096e5c4eb1e9677d2"
server_key_iv: "4a1edd12ede88b3abe466bed8a96be3d"
server_key_version: "encryptionkey@0"
I20260927 15:06:46.035840 22304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 15:06:46.038412 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:46.039772 22304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:46.039937 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "806576f461bd4d77a0261dd872b9c698"
format_stamp: "Formatted at 2026-09-27 15:06:46 on dist-test-slave-ml7t"
server_key: "5b610b172526f76096e5c4eb1e9677d2"
server_key_iv: "4a1edd12ede88b3abe466bed8a96be3d"
server_key_version: "encryptionkey@0"
I20260927 15:06:46.040059 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:46.056506 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:46.059875 22304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:46.060097 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:46.067395 22304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35125
I20260927 15:06:46.067423 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35125 every 8 connection(s)
I20260927 15:06:46.068374 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:46.071267 22372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:46.076800 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698: Bootstrap starting.
I20260927 15:06:46.077250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22304
I20260927 15:06:46.077400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:46.077666 392 external_mini_cluster.cc:1428] Setting key 714b213d0f0cdd4abccfeec134bc5df8
I20260927 15:06:46.079701 22372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:46.080482 22372 log.cc:824] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:46.082983 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698: No bootstrap required, opened a new log
Sep 27 15:06:46 dist-test-slave-ml7t krb5kdc[22291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:46.088322 22372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806576f461bd4d77a0261dd872b9c698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260927 15:06:46.088634 22372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:46.088707 22372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806576f461bd4d77a0261dd872b9c698, State: Initialized, Role: FOLLOWER
I20260927 15:06:46.089298 22372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806576f461bd4d77a0261dd872b9c698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260927 15:06:46.089484 22372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:46.089558 22372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:46.089677 22372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:46.090720 22372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806576f461bd4d77a0261dd872b9c698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260927 15:06:46.091089 22372 leader_election.cc:302] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 806576f461bd4d77a0261dd872b9c698; no voters:
I20260927 15:06:46.091456 22372 leader_election.cc:288] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:46.091530 22377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:46.091747 22377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [term 1 LEADER]: Becoming Leader. State: Replica: 806576f461bd4d77a0261dd872b9c698, State: Running, Role: LEADER
I20260927 15:06:46.092139 22377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806576f461bd4d77a0261dd872b9c698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260927 15:06:46.092896 22372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:46.093962 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "806576f461bd4d77a0261dd872b9c698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806576f461bd4d77a0261dd872b9c698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } } }
I20260927 15:06:46.094091 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:46.094100 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:46.079138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52269 (local address 127.0.98.62:35125)
0927 15:06:46.079604 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:46.079619 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:46.079657 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:06:46.080407 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:46.080427 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:46.080880 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:46.081325 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:46.082371 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:46.083461 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:46.084687 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:46.085190 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:46.085334 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:46.088652 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:46.088660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:46.088681 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:46.088722 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:46.090667 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:46.091178 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:46.091184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:46.091191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:46.091369 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:46.091674 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:46.091677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:46.091679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:46.092241 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:46.092437 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:46.092726 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:06:46.093072 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":146,"threads_started":1}
I20260927 15:06:46.094679 22378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806576f461bd4d77a0261dd872b9c698. Latest consensus state: current_term: 1 leader_uuid: "806576f461bd4d77a0261dd872b9c698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806576f461bd4d77a0261dd872b9c698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } } }
I20260927 15:06:46.094856 22378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:46.095243 22386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:46.097424 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:46.098450 22386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:46.105360 22386 catalog_manager.cc:1347] Generated new cluster ID: daa9337cc1e94e68813c594a895b1285
I20260927 15:06:46.105443 22386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:46.108265 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:46.099706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46579 (local address 127.0.98.62:35125)
0927 15:06:46.099820 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:46.099824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:46.099969 (+ 145us) server_negotiation.cc:407] Connection header received
0927 15:06:46.100117 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:46.100121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:46.100171 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:46.100259 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:46.100967 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:46.101514 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:46.102408 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:46.102668 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:46.102783 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:46.104494 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:46.104498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:46.104501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:46.104529 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:46.106288 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:46.106821 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:46.106825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:46.106828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:46.106918 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:46.107237 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:46.107241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:46.107243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:46.107466 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:46.107567 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:46.107972 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 15:06:46.108100 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 15:06:46.115073 22386 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:46.116601 22386 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:46.121803 22386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 806576f461bd4d77a0261dd872b9c698: Generated new TSK 0
I20260927 15:06:46.122598 22386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:46.132841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22304
2026-09-27T15:06:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:46 dist-test-slave-ml7t krb5kdc[22403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:46 dist-test-slave-ml7t krb5kdc[22403](info): set up 2 sockets
Sep 27 15:06:46 dist-test-slave-ml7t krb5kdc[22403](info): commencing operation
krb5kdc: starting...
W20260927 15:06:47.194270 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:47 dist-test-slave-ml7t krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:47.317128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:47.425843 22416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:47.426120 22416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:47.426163 22416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:47.430848 22416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:47.430917 22416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:47.430936 22416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:47.430954 22416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:47.430969 22416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:47.436173 22416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:47.437177 22416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:47.438154 22416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:47.446460 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:47.446506 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:47.446506 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:47.446805 22416 server_base.cc:1034] running on GCE node
I20260927 15:06:47.447451 22416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:47.448560 22416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:47.449760 22416 hybrid_clock.cc:648] HybridClock initialized: now 1790521607449739 us; error 34 us; skew 500 ppm
Sep 27 15:06:47 dist-test-slave-ml7t krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:47.452514 22416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:47.453388 22416 webserver.cc:460] Webserver started at http://127.0.98.62:43989/ using document root <none> and password file <none>
I20260927 15:06:47.454006 22416 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:47.454095 22416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:47.454326 22416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:47.456300 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "d4a505b9aaac47e49baefe20654ecc0b"
format_stamp: "Formatted at 2026-09-27 15:06:47 on dist-test-slave-ml7t"
server_key: "42191cf1335b96c267ccb8aba38d2604"
server_key_iv: "2c74cfc80dd766fcb320cd8b1da62ec3"
server_key_version: "encryptionkey@0"
I20260927 15:06:47.456851 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4a505b9aaac47e49baefe20654ecc0b"
format_stamp: "Formatted at 2026-09-27 15:06:47 on dist-test-slave-ml7t"
server_key: "42191cf1335b96c267ccb8aba38d2604"
server_key_iv: "2c74cfc80dd766fcb320cd8b1da62ec3"
server_key_version: "encryptionkey@0"
I20260927 15:06:47.460475 22416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 15:06:47.463090 22431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:47.464366 22416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:47.464512 22416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "d4a505b9aaac47e49baefe20654ecc0b"
format_stamp: "Formatted at 2026-09-27 15:06:47 on dist-test-slave-ml7t"
server_key: "42191cf1335b96c267ccb8aba38d2604"
server_key_iv: "2c74cfc80dd766fcb320cd8b1da62ec3"
server_key_version: "encryptionkey@0"
I20260927 15:06:47.464629 22416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:47.492161 22416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:47.495265 22416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:47.495478 22416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:47.502382 22416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32983
I20260927 15:06:47.502411 22483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32983 every 8 connection(s)
I20260927 15:06:47.503362 22416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:47.503674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22416
I20260927 15:06:47.503841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:47.504142 392 external_mini_cluster.cc:1428] Setting key 683336db1971bce84de6928189a70c2e
I20260927 15:06:47.506551 22484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:47.513029 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b: Bootstrap starting.
I20260927 15:06:47.513691 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:47.505651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56199 (local address 127.0.98.62:32983)
0927 15:06:47.506130 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:47.506152 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:47.506187 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:06:47.506915 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:47.506934 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:47.507287 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:47.507569 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:47.508061 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:47.509076 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:47.510362 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:47.510823 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:47.510939 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:47.511874 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:47.511880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:47.511893 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:47.512289 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:47.512344 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:47.512367 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:47.512443 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:47.512612 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 15:06:47.512845 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":203,"threads_started":1}
I20260927 15:06:47.515308 22484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:47.516165 22484 log.cc:824] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:47.516472 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:47.518522 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b: No bootstrap required, opened a new log
I20260927 15:06:47.522630 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:47.518619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58387 (local address 127.0.98.62:32983)
0927 15:06:47.518736 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:47.518740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:47.518943 (+ 203us) server_negotiation.cc:407] Connection header received
0927 15:06:47.519029 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:47.519032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:47.519078 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:47.519169 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:47.519411 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:47.520151 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:47.520952 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:47.521329 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:47.521428 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:47.522232 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:47.522236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:47.522239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:47.522251 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:47.522282 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:47.522295 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:47.522369 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:47.522442 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:06:47.522501 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:06:47.523236 22484 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a505b9aaac47e49baefe20654ecc0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32983 } }
I20260927 15:06:47.523495 22484 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:47.523564 22484 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a505b9aaac47e49baefe20654ecc0b, State: Initialized, Role: FOLLOWER
I20260927 15:06:47.523968 22484 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a505b9aaac47e49baefe20654ecc0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32983 } }
I20260927 15:06:47.524115 22484 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:47.524201 22484 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:47.524369 22484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:47.525260 22484 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a505b9aaac47e49baefe20654ecc0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32983 } }
I20260927 15:06:47.525591 22484 leader_election.cc:302] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a505b9aaac47e49baefe20654ecc0b; no voters:
I20260927 15:06:47.525938 22484 leader_election.cc:288] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:47.526073 22494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:47.526291 22494 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [term 1 LEADER]: Becoming Leader. State: Replica: d4a505b9aaac47e49baefe20654ecc0b, State: Running, Role: LEADER
I20260927 15:06:47.526636 22494 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a505b9aaac47e49baefe20654ecc0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32983 } }
I20260927 15:06:47.527236 22484 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:47.528429 22495 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a505b9aaac47e49baefe20654ecc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a505b9aaac47e49baefe20654ecc0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32983 } } }
I20260927 15:06:47.528559 22495 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:47.528877 22496 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a505b9aaac47e49baefe20654ecc0b. Latest consensus state: current_term: 1 leader_uuid: "d4a505b9aaac47e49baefe20654ecc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a505b9aaac47e49baefe20654ecc0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32983 } } }
I20260927 15:06:47.528975 22503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:47.529021 22496 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:47.531733 22503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:47.536767 22503 catalog_manager.cc:1347] Generated new cluster ID: 92ca0326fb374a3988a747f58895120a
I20260927 15:06:47.536841 22503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:47.564385 22503 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:47.565749 22503 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:47.573971 22503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a505b9aaac47e49baefe20654ecc0b: Generated new TSK 0
I20260927 15:06:47.574857 22503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:47.589998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22416
2026-09-27T15:06:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:47 dist-test-slave-ml7t krb5kdc[22515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:47 dist-test-slave-ml7t krb5kdc[22515](info): set up 2 sockets
Sep 27 15:06:47 dist-test-slave-ml7t krb5kdc[22515](info): commencing operation
krb5kdc: starting...
W20260927 15:06:48.626216 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:48 dist-test-slave-ml7t krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:48.749456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:48.857133 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:48.857420 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:48.857465 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:48.861531 22528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:48.861600 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:48.861621 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:48.861639 22528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:48.861653 22528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:48.867098 22528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:48.868180 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:48.869203 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:06:48.877794 22528 server_base.cc:1034] running on GCE node
W20260927 15:06:48.877723 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:48.877696 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:48.877718 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:48.878544 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:48.879659 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:48.880859 22528 hybrid_clock.cc:648] HybridClock initialized: now 1790521608880845 us; error 25 us; skew 500 ppm
Sep 27 15:06:48 dist-test-slave-ml7t krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:48.883761 22528 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:48.884676 22528 webserver.cc:460] Webserver started at http://127.0.98.62:41663/ using document root <none> and password file <none>
I20260927 15:06:48.885252 22528 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:48.885342 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:48.885568 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:48.887624 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "696b57a349f441739f56503b4d17b508"
format_stamp: "Formatted at 2026-09-27 15:06:48 on dist-test-slave-ml7t"
server_key: "f40d6021969935d11f6a7031d6c07d6b"
server_key_iv: "02398db1af6ba87967c76cb3bf507e38"
server_key_version: "encryptionkey@0"
I20260927 15:06:48.888181 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "696b57a349f441739f56503b4d17b508"
format_stamp: "Formatted at 2026-09-27 15:06:48 on dist-test-slave-ml7t"
server_key: "f40d6021969935d11f6a7031d6c07d6b"
server_key_iv: "02398db1af6ba87967c76cb3bf507e38"
server_key_version: "encryptionkey@0"
I20260927 15:06:48.891829 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:06:48.894256 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:48.895485 22528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:48.895620 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "696b57a349f441739f56503b4d17b508"
format_stamp: "Formatted at 2026-09-27 15:06:48 on dist-test-slave-ml7t"
server_key: "f40d6021969935d11f6a7031d6c07d6b"
server_key_iv: "02398db1af6ba87967c76cb3bf507e38"
server_key_version: "encryptionkey@0"
I20260927 15:06:48.895731 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:48.900137 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:48.903088 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:48.903307 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:48.910331 22528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38747
I20260927 15:06:48.910351 22595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38747 every 8 connection(s)
I20260927 15:06:48.911278 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:48.914057 22596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:48.915581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22528
I20260927 15:06:48.915721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:48.916000 392 external_mini_cluster.cc:1428] Setting key de274a0bbcb31ffb35405a1bfcea5741
I20260927 15:06:48.920295 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508: Bootstrap starting.
I20260927 15:06:48.921277 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:48.917503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35899 (local address 127.0.98.62:38747)
0927 15:06:48.917976 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:48.917992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:48.918028 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:06:48.918658 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:48.918675 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:48.918969 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:48.919307 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:48.920103 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:48.920107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:48.920120 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:48.920130 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:48.920181 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:48.920200 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:48.920346 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:48.920416 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:06:48.920573 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":134,"threads_started":1}
I20260927 15:06:48.922750 22596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:48.923640 22596 log.cc:824] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:48.924147 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:48.925981 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508: No bootstrap required, opened a new log
I20260927 15:06:48.928006 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:48.926452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38283 (local address 127.0.98.62:38747)
0927 15:06:48.926570 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:48.926574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:48.926753 (+ 179us) server_negotiation.cc:407] Connection header received
0927 15:06:48.926847 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:48.926850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:48.926882 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:48.926967 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:48.927631 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:48.927636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:48.927640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:06:48.927652 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:48.927688 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:06:48.927700 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:48.927785 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:48.927856 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 15:06:48.927892 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:06:48.930086 22596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696b57a349f441739f56503b4d17b508" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38747 } }
I20260927 15:06:48.930299 22596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:06:48.930366 22596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696b57a349f441739f56503b4d17b508, State: Initialized, Role: FOLLOWER
I20260927 15:06:48.930807 22596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696b57a349f441739f56503b4d17b508" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38747 } }
I20260927 15:06:48.930949 22596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:06:48.931016 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:06:48.931116 22596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:48.932034 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696b57a349f441739f56503b4d17b508" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38747 } }
I20260927 15:06:48.932375 22596 leader_election.cc:302] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 696b57a349f441739f56503b4d17b508; no voters:
I20260927 15:06:48.932669 22596 leader_election.cc:288] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:06:48.932813 22606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:48.933043 22606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [term 1 LEADER]: Becoming Leader. State: Replica: 696b57a349f441739f56503b4d17b508, State: Running, Role: LEADER
I20260927 15:06:48.933393 22606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696b57a349f441739f56503b4d17b508" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38747 } }
I20260927 15:06:48.933954 22596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:48.935232 22608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 696b57a349f441739f56503b4d17b508. Latest consensus state: current_term: 1 leader_uuid: "696b57a349f441739f56503b4d17b508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696b57a349f441739f56503b4d17b508" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38747 } } }
I20260927 15:06:48.935357 22608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:48.935308 22607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "696b57a349f441739f56503b4d17b508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696b57a349f441739f56503b4d17b508" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38747 } } }
I20260927 15:06:48.935413 22607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:48.935719 22615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:48.938890 22615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:48.945364 22615 catalog_manager.cc:1347] Generated new cluster ID: f396e4d29041498dbcf6b490b5ba2c7a
I20260927 15:06:48.945448 22615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:48.975481 22615 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:48.975927 22615 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:48.985909 22615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 696b57a349f441739f56503b4d17b508: Generated new TSK 0
I20260927 15:06:48.986676 22615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:06:48.994077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22528
2026-09-27T15:06:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1403 ms)
[----------] 18 tests from AuthTokenIssuingTest (25696 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:49 dist-test-slave-ml7t krb5kdc[22627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:49 dist-test-slave-ml7t krb5kdc[22627](info): set up 2 sockets
Sep 27 15:06:49 dist-test-slave-ml7t krb5kdc[22627](info): commencing operation
krb5kdc: starting...
W20260927 15:06:50.082087 392 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:50.205425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40395,127.0.98.61:41047,127.0.98.60:39905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:50.316238 22640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:06:50.316517 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:50.316540 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:50.316576 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:50.320438 22640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:50.320492 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:50.320510 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:50.320526 22640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:50.320541 22640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:50.325554 22640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40395,127.0.98.61:41047,127.0.98.60:39905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:50.326560 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:50.327500 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:50.335577 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:50.335644 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:50.335580 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:50.335850 22640 server_base.cc:1034] running on GCE node
I20260927 15:06:50.336599 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:50.337878 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:50.339035 22640 hybrid_clock.cc:648] HybridClock initialized: now 1790521610339006 us; error 42 us; skew 500 ppm
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:50.345624 22640 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:50.346585 22640 webserver.cc:460] Webserver started at http://127.0.98.62:41459/ using document root <none> and password file <none>
I20260927 15:06:50.347136 22640 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:50.347263 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:50.347492 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:50.349453 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "8e29171875d043808d0789ac4c374f7d"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "bcda561e4425fd248bdc37672776ac47"
server_key_iv: "43b6055a258017cddc9ab61f9f000e89"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.350045 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e29171875d043808d0789ac4c374f7d"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "bcda561e4425fd248bdc37672776ac47"
server_key_iv: "43b6055a258017cddc9ab61f9f000e89"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.353652 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:06:50.356035 22655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:50.357268 22640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:06:50.357409 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "8e29171875d043808d0789ac4c374f7d"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "bcda561e4425fd248bdc37672776ac47"
server_key_iv: "43b6055a258017cddc9ab61f9f000e89"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.357515 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:50.378849 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:50.382126 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:50.382321 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:50.389010 22640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40395
I20260927 15:06:50.389277 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40395 every 8 connection(s)
I20260927 15:06:50.390070 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:50.391691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22640
I20260927 15:06:50.391840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:50.392107 392 external_mini_cluster.cc:1428] Setting key 96f07c346e0fd70ea1f61d4d0d5c866d
I20260927 15:06:50.393181 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:50.397221 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:50.412618 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.399573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.0.1:53366)
0927 15:06:50.400182 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.400217 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.401067 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.401871 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.401876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.402218 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.402519 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.402533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.403711 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.403714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.404437 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.404445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.405072 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.405075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.405181 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.405864 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.405879 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.407573 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:50.409504 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.409514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.409526 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.409748 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.410089 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.410091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.410093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.410207 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.410836 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:50.410842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:50.410981 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 15:06:50.411144 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:06:50.411491 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"spinlock_wait_cycles":107904,"thread_start_us":178,"threads_started":1}
I20260927 15:06:50.412618 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.399572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.98.62:40395)
0927 15:06:50.400032 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.400051 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.400323 (+ 272us) server_negotiation.cc:407] Connection header received
0927 15:06:50.401380 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.401384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.401600 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.401842 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.402667 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.403582 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.404552 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.404921 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.405088 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.407738 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.407744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.407754 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.407773 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.409395 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.409878 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.409886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.409892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.409990 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.410303 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.410306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.410307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.410689 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.410852 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.411219 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.411566 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":318,"thread_start_us":174,"threads_started":1}
I20260927 15:06:50.415169 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:50.415885 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.415648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42582)
0927 15:06:50.415759 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.415825 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:06:50.416080 22658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
I20260927 15:06:50.416893 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.416689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42584)
0927 15:06:50.416795 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.416841 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:06:50.417021 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41047: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 15:06:50.418565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41047
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40395,127.0.98.61:41047,127.0.98.60:39905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:06:50.466269 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } attempt: 1
I20260927 15:06:50.467206 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.466836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42586)
0927 15:06:50.467007 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.467072 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 15:06:50.467941 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.467749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42600)
0927 15:06:50.467842 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.467886 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:06:50.468091 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41047: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
W20260927 15:06:50.530580 22715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:06:50.530846 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:50.530869 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:50.530972 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:50.535000 22715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:50.535064 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:50.535084 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:50.535100 22715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:50.535138 22715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:50.536326 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } attempt: 2
I20260927 15:06:50.537230 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.536911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42616)
0927 15:06:50.537078 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.537153 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:06:50.538100 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.537914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42624)
0927 15:06:50.538006 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.538049 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:06:50.538249 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41047: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41047: connect: Connection refused (error 111)
I20260927 15:06:50.541178 22715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40395,127.0.98.61:41047,127.0.98.60:39905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:50.542361 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:50.543387 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:50.551864 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:50.551864 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:50.551870 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:50.552031 22715 server_base.cc:1034] running on GCE node
I20260927 15:06:50.552681 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:50.553958 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:50.555155 22715 hybrid_clock.cc:648] HybridClock initialized: now 1790521610555135 us; error 31 us; skew 500 ppm
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:50.558104 22715 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 15:06:50.558996 22715 webserver.cc:460] Webserver started at http://127.0.98.61:40389/ using document root <none> and password file <none>
I20260927 15:06:50.559576 22715 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:50.559639 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:50.559806 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:50.562077 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data/instance:
uuid: "61800bfb9c7a46e1b69ce3ace2b01718"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "24de6f2060ec4e33868faeb1229e31d1"
server_key_iv: "678966cc233cd36f65427814ee7467fb"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.562584 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/wal/instance:
uuid: "61800bfb9c7a46e1b69ce3ace2b01718"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "24de6f2060ec4e33868faeb1229e31d1"
server_key_iv: "678966cc233cd36f65427814ee7467fb"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.566275 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:06:50.568668 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:50.570043 22715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:06:50.570218 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/wal
uuid: "61800bfb9c7a46e1b69ce3ace2b01718"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "24de6f2060ec4e33868faeb1229e31d1"
server_key_iv: "678966cc233cd36f65427814ee7467fb"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.570312 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:50.582696 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:50.585947 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:50.586159 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:50.593127 22715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41047
I20260927 15:06:50.593415 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41047 every 8 connection(s)
I20260927 15:06:50.594168 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
I20260927 15:06:50.594408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22715
I20260927 15:06:50.594532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-1/wal/instance
I20260927 15:06:50.594878 392 external_mini_cluster.cc:1428] Setting key 0ef4450a4ac66419aca5849b08b41bfb
I20260927 15:06:50.597751 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:50.603190 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:50.618114 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.605290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.0.98.62:40395)
0927 15:06:50.605445 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.605450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.606474 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 15:06:50.607515 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.607522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.607592 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.607700 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.609249 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.610174 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.611363 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.611840 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.611980 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.614345 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.614348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.614350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.614376 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.615766 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.616506 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.616509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.616510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.616584 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.616975 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.616977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.616978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.617164 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.617249 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.617747 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.617936 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:06:50.618892 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.605817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.0.1:53380)
0927 15:06:50.606309 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.606365 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.607303 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.607924 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.607935 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.608403 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.609093 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.609109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.610334 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.610337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.611220 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.611230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.611986 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.611989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.612121 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.612762 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.612778 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.614166 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:50.615887 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.615892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.615904 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.616404 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.616692 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.616695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.616696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.616882 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.617282 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:50.617288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:50.617514 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 15:06:50.617671 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:06:50.617985 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260927 15:06:50.620294 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 15:06:50.622337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39905
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40395,127.0.98.61:41047,127.0.98.60:39905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:06:50.629431 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } attempt: 3
I20260927 15:06:50.630733 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.620766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42636)
0927 15:06:50.620874 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.620889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.620957 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.621747 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.621751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.621978 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.622092 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.622100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.623132 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.623139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.623887 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.623896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.624748 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.624750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.624884 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.625380 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.625400 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.626917 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:50.628774 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.628780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.628784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.629051 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.629365 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.629370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.629374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.629541 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.630216 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:50.630218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:50.630305 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 15:06:50.630418 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:06:50.630549 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":32}
I20260927 15:06:50.631052 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.620923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.0.98.61:41047)
0927 15:06:50.621292 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.621305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.621330 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:06:50.621375 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.621384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.621576 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.621717 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.622213 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.623022 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.624034 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.624624 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.624785 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.627049 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.627057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.627067 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.627086 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.628658 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.629156 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.629160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.629162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.629251 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.629665 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.629670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.629683 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.630041 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.630175 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.630547 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.630727 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":248,"thread_start_us":139,"threads_started":1}
I20260927 15:06:50.633222 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:50.633946 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.633661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56008)
0927 15:06:50.633802 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.633881 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:06:50.634454 22734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:06:50.635282 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.635072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56024)
0927 15:06:50.635145 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.635225 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 15:06:50.635426 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
I20260927 15:06:50.638952 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.629964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42638)
0927 15:06:50.630170 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.630186 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.630272 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.630583 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.630586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.630755 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.630878 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.630888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.631934 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.631938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.632942 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.632958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.633497 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.633500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.633602 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.634123 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.634135 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.635503 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:50.637406 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.637408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.637410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.637638 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.637989 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.637992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.637993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.638066 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.638443 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:50.638446 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:50.638541 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 15:06:50.638644 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:06:50.638802 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 15:06:50.638965 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.630021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.0.98.61:41047)
0927 15:06:50.630351 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.630360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.630374 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:06:50.630408 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.630413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.630450 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.630998 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.631078 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.631791 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.633071 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.633374 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.633476 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.635689 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.635693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.635695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.635719 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.637295 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.637758 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.637761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.637763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.637877 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.638162 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.638165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.638167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.638346 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.638451 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.638704 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.638805 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":97,"threads_started":1}
I20260927 15:06:50.639691 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:50.640352 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.640111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56040)
0927 15:06:50.640229 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.640283 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:06:50.641209 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.640925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56056)
0927 15:06:50.641071 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.641128 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 15:06:50.641359 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
I20260927 15:06:50.672538 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 1
I20260927 15:06:50.673425 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.673081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56066)
0927 15:06:50.673254 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.673331 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:06:50.674322 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.674076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56072)
0927 15:06:50.674181 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.674256 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:06:50.674520 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
I20260927 15:06:50.684695 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 1
I20260927 15:06:50.685421 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.685135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56078)
0927 15:06:50.685279 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.685341 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 15:06:50.686143 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.685919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56088)
0927 15:06:50.686030 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.686082 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:06:50.686299 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
W20260927 15:06:50.743002 22789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:06:50.743297 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:50.743324 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:50.743366 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:50.747648 22789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:50.747700 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:50.747721 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:50.747740 22789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:50.747754 22789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:50.749694 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 2
I20260927 15:06:50.750578 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.750255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56096)
0927 15:06:50.750432 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.750499 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 15:06:50.751353 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.751101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56112)
0927 15:06:50.751240 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.751292 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 15:06:50.751550 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
I20260927 15:06:50.753923 22789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40395,127.0.98.61:41047,127.0.98.60:39905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:50.754457 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 2
I20260927 15:06:50.755152 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.754877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56114)
0927 15:06:50.755017 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.755078 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:06:50.755240 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:50.755919 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.755682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56122)
0927 15:06:50.755804 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.755856 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:06:50.756057 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
I20260927 15:06:50.756464 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:50.765030 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:50.765055 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:50.765043 22796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:50.765388 22789 server_base.cc:1034] running on GCE node
I20260927 15:06:50.766002 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:50.767069 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:50.768251 22789 hybrid_clock.cc:648] HybridClock initialized: now 1790521610768218 us; error 46 us; skew 500 ppm
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:50.771340 22789 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 15:06:50.772477 22789 webserver.cc:460] Webserver started at http://127.0.98.60:41205/ using document root <none> and password file <none>
I20260927 15:06:50.773197 22789 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:50.773461 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:50.773866 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:50.776346 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data/instance:
uuid: "755d9186b2cb40b09f52a013ed2c6bec"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "ccee99cffc2538f8a60a688da4c79fc2"
server_key_iv: "3f859d9bf61daab9aad7681b5c346772"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.776959 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/wal/instance:
uuid: "755d9186b2cb40b09f52a013ed2c6bec"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "ccee99cffc2538f8a60a688da4c79fc2"
server_key_iv: "3f859d9bf61daab9aad7681b5c346772"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.780988 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:06:50.783651 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:50.785159 22789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:06:50.785362 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/wal
uuid: "755d9186b2cb40b09f52a013ed2c6bec"
format_stamp: "Formatted at 2026-09-27 15:06:50 on dist-test-slave-ml7t"
server_key: "ccee99cffc2538f8a60a688da4c79fc2"
server_key_iv: "3f859d9bf61daab9aad7681b5c346772"
server_key_version: "encryptionkey@0"
I20260927 15:06:50.785524 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:50.844208 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:50.847229 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 3
I20260927 15:06:50.847941 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:50.848012 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.847731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56132)
0927 15:06:50.847875 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.847928 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:06:50.848161 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:50.848809 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.848521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56142)
0927 15:06:50.848686 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.848728 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 15:06:50.848968 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
I20260927 15:06:50.850713 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 3
I20260927 15:06:50.851435 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.851130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56156)
0927 15:06:50.851274 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.851333 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:06:50.852210 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.851953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56172)
0927 15:06:50.852066 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.852128 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 15:06:50.852401 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39905: connect: Connection refused (error 111)
I20260927 15:06:50.855963 22789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39905
I20260927 15:06:50.856307 22858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39905 every 8 connection(s)
I20260927 15:06:50.856986 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
I20260927 15:06:50.859373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 22789
I20260927 15:06:50.859530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521328452215-392-0/minicluster-data/master-2/wal/instance
I20260927 15:06:50.859889 392 external_mini_cluster.cc:1428] Setting key e6c4b3e5d60f12d28c2042a78eedb5e8
I20260927 15:06:50.860466 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:50.865566 22859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:50.875633 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.861309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.0.98.60:39905)
0927 15:06:50.861822 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.861852 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.861898 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:06:50.862785 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.862809 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.863255 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.863660 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.864378 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.865900 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.866854 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.867405 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.867596 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.870392 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.870399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.870410 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.870439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.872398 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.872882 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.872889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.872909 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.873079 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.873370 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.873375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.873377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.873861 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.874102 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.874440 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.874774 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260927 15:06:50.877437 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.867538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.0.98.62:40395)
0927 15:06:50.867669 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.867674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.868267 (+ 593us) server_negotiation.cc:407] Connection header received
0927 15:06:50.868403 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.868407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.868470 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.868560 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.869200 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.869731 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.870871 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.871289 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.871390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.873816 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.873822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.873827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.873857 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.875537 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.876095 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.876098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.876099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.876177 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.876500 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.876503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.876504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.876729 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.876816 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.877192 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.877317 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:06:50.877442 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.867474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.0.1:53394)
0927 15:06:50.868131 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.868191 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.868304 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.868607 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.868611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.868944 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.869064 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.869070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.869891 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.869894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.870726 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.870737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.871401 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.871404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.871525 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.871989 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.871999 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.873657 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:50.875654 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.875664 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.875670 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.875983 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.876276 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.876279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.876280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.876370 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.876834 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:50.876839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:50.876952 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 15:06:50.877095 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 15:06:50.877260 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":100,"threads_started":1}
I20260927 15:06:50.878857 22859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:50.887409 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.879382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41047 (local address 127.0.0.1:42652)
0927 15:06:50.879484 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.879503 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.879574 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.879823 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.879826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.879981 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.880073 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.880078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.880839 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.880845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.881348 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.881355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.881892 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.881895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.881996 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.882404 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.882415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.883652 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:50.885622 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.885629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.885633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.885894 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.886237 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.886240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.886247 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.886341 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.886847 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:50.886850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:50.886947 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:06:50.887055 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:06:50.887232 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 15:06:50.887419 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.879457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.61:41047)
0927 15:06:50.879576 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.879581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.879595 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:06:50.879671 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.879674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.879716 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.879810 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.880195 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.880732 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.881464 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.881745 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.881893 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.883801 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.883805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.883808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.883841 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.885492 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.886003 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.886006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.886007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.886081 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.886471 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.886475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.886477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.886732 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.886857 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.887131 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.887304 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:06:50.888257 22859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:50.888355 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.878465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.0.98.62:40395)
0927 15:06:50.878573 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.878577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.878671 (+ 94us) server_negotiation.cc:407] Connection header received
0927 15:06:50.878783 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.878786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.878829 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.878911 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.879691 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.880197 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.881000 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.881238 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.881350 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.884459 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.884463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.884465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.884498 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.886335 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.886847 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.886850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.886852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.886937 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.887257 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.887261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.887262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.887468 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.887703 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.888047 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.888187 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:06:50.897432 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.889022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56186)
0927 15:06:50.889142 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.889157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.889271 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.889554 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.889556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.889709 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.889825 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.889831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.890641 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.890647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.891149 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.891156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.891611 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.891615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.891710 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.892226 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.892239 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.893728 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:50.895741 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.895747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.895750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.895994 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.896317 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:50.896323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:50.896326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:50.896419 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:50.896868 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:50.896870 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:50.896975 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 15:06:50.897102 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 15:06:50.897245 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":5}
I20260927 15:06:50.897516 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.889179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.0.98.60:39905)
0927 15:06:50.889281 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.889286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.889299 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:06:50.889355 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.889358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.889393 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.889457 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.889954 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.890535 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.891258 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.891509 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.891652 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.893895 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.893899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.893901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.893922 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.895583 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.896105 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.896108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.896109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.896215 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.896514 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.896516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.896517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.896767 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.896872 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.897227 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.897356 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":45}
I20260927 15:06:50.898360 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.889545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.0.98.61:41047)
0927 15:06:50.889648 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.889652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.889712 (+ 60us) server_negotiation.cc:407] Connection header received
0927 15:06:50.889842 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.889845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.889898 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.889964 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.890674 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.891148 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.891904 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.892146 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.892258 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.894839 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.894845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.894848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.894876 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:50.896513 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.896968 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.896972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.896974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.897067 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:50.897350 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:50.897353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:50.897355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:50.897585 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:50.897809 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:50.898033 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:06:50.898147 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 15:06:50.901937 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:50.902215 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: Bootstrap starting.
I20260927 15:06:50.903293 22682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8e29171875d043808d0789ac4c374f7d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53404
I20260927 15:06:50.904734 22859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:50.905495 22859 log.cc:824] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:50.907514 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: No bootstrap required, opened a new log
I20260927 15:06:50.910593 22859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:50.910835 22859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d9186b2cb40b09f52a013ed2c6bec, State: Initialized, Role: FOLLOWER
I20260927 15:06:50.911248 22859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:50.911758 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:50.911892 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:50.912658 22859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:06:50.917583 22876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:50.917677 22876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:50.916751 22789 master_runner.cc:186] Error getting master registration for 127.0.98.62:40395: OK, instance_id { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" instance_seqno: 1790521610387143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:06:50.920692 22789 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:06:50.992094 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 4
Sep 27 15:06:50 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:06:51.002544 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.992751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56192)
0927 15:06:50.992921 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:50.992938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:50.993050 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:50.993475 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:50.993480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:50.993702 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:50.993888 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.993896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.994970 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.994974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.995826 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:50.995837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.996453 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.996456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:50.996549 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.997136 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:50.997151 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:50.998836 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:51.000857 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:51.000859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:51.000861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:51.001056 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:51.001394 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:51.001397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:51.001399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:51.001514 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:51.001988 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:51.001993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:51.002100 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 15:06:51.002205 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 15:06:51.002353 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 15:06:51.002557 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:50.992834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.0.98.60:39905)
0927 15:06:50.992965 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:50.992971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:50.993008 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:06:50.993188 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:50.993191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:50.993263 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:50.993339 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:50.994036 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.994837 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.995964 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:50.996326 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:50.996435 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:50.999013 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:50.999017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:50.999020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:50.999049 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:51.000757 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.001161 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.001164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.001166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.001266 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.001616 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.001619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.001622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.001880 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:51.001994 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:51.002275 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:06:51.002398 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:06:51.006278 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: Bootstrap starting.
I20260927 15:06:51.009254 22783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:51.010128 22783 log.cc:824] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:51.012315 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: No bootstrap required, opened a new log
I20260927 15:06:51.016512 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } attempt: 4
I20260927 15:06:51.016753 22783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:51.017014 22783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61800bfb9c7a46e1b69ce3ace2b01718, State: Initialized, Role: FOLLOWER
I20260927 15:06:51.017380 22783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:51.018056 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.018307 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:51.019033 22783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 15:06:51 dist-test-slave-ml7t krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:06:51.023278 22715 master_runner.cc:186] Error getting master registration for 127.0.98.62:40395: OK, instance_id { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" instance_seqno: 1790521610387143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:06:51.027318 22715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:06:51.027487 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:51.017104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39905 (local address 127.0.0.1:56196)
0927 15:06:51.017264 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:51.017282 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:51.017400 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:51.017955 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:51.017958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:51.018191 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:51.018324 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:51.018335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.019591 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.019595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:51.020495 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:51.020508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.021136 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.021144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:51.021263 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:51.021930 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:51.021950 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:51.023605 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:51.025740 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:51.025743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:51.025746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:51.026031 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:51.026360 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:51.026363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:51.026365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:51.026469 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:51.026936 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:51.026939 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:51.027038 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 15:06:51.027156 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 15:06:51.027305 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:06:51.028160 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:51.017469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.0.98.60:39905)
0927 15:06:51.017630 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.017634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.017650 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:06:51.017700 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.017703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.017750 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.018489 (+ 739us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.018594 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.019403 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.020620 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.021001 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.021115 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:51.023769 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:51.023773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:51.023777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:51.023806 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:51.025610 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.026164 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.026167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.026169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.026258 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.026576 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.026579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.026581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.026797 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:51.026890 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:51.027941 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 15:06:51.028043 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 15:06:51.030297 22890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:51.030381 22890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:51.031420 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d: Bootstrap starting.
I20260927 15:06:51.033907 22708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:51.035012 22708 log.cc:824] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:51.037511 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d: No bootstrap required, opened a new log
I20260927 15:06:51.040205 22708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:51.040452 22708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e29171875d043808d0789ac4c374f7d, State: Initialized, Role: FOLLOWER
I20260927 15:06:51.040838 22708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:51.041369 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.041533 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:51.042337 22708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:51.042478 22682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:06:51.042675 22682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:51.044127 22682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:51.045380 22682 leader_election.cc:288] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [CANDIDATE]: Term 1 election: Requested vote from peers 61800bfb9c7a46e1b69ce3ace2b01718 (127.0.98.61:41047), 755d9186b2cb40b09f52a013ed2c6bec (127.0.98.60:39905)
I20260927 15:06:51.046304 22833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e29171875d043808d0789ac4c374f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "755d9186b2cb40b09f52a013ed2c6bec"
I20260927 15:06:51.046584 22833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:51.048133 22833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e29171875d043808d0789ac4c374f7d in term 1.
I20260927 15:06:51.048938 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e29171875d043808d0789ac4c374f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61800bfb9c7a46e1b69ce3ace2b01718"
I20260927 15:06:51.049213 22757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:51.050712 22757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e29171875d043808d0789ac4c374f7d in term 1.
I20260927 15:06:51.051200 22659 leader_election.cc:302] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d9186b2cb40b09f52a013ed2c6bec, 8e29171875d043808d0789ac4c374f7d; no voters:
I20260927 15:06:51.051678 22893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:51.051921 22893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [term 1 LEADER]: Becoming Leader. State: Replica: 8e29171875d043808d0789ac4c374f7d, State: Running, Role: LEADER
I20260927 15:06:51.052289 22893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } }
I20260927 15:06:51.054193 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e29171875d043808d0789ac4c374f7d. Latest consensus state: current_term: 1 leader_uuid: "8e29171875d043808d0789ac4c374f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.054315 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:51.055259 22640 master_runner.cc:263] No leader master found from master 8e29171875d043808d0789ac4c374f7d
I20260927 15:06:51.055351 22640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:06:51.058257 22904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:51.058339 22904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:51.060153 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:51.047739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.98.62:40395)
0927 15:06:51.047859 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.047864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.049528 (+ 1664us) server_negotiation.cc:407] Connection header received
0927 15:06:51.049639 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.049643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.049696 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.049817 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.050769 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.051555 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.052808 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.053138 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.053229 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:51.055344 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:51.055348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:51.055351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:51.055381 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:51.057813 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.058670 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.058674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.058676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.058768 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.059081 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.059084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.059086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.059331 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:51.059451 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:51.059836 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 15:06:51.059969 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:06:51.059979 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:51.061333 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:51.067324 22833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [term 1 FOLLOWER]: Refusing update from remote peer 8e29171875d043808d0789ac4c374f7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:51.068084 22912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:51.072567 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e29171875d043808d0789ac4c374f7d. Latest consensus state: current_term: 1 leader_uuid: "8e29171875d043808d0789ac4c374f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.072736 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:51.073997 22757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [term 1 FOLLOWER]: Refusing update from remote peer 8e29171875d043808d0789ac4c374f7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:51.074710 22912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:51.079118 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e29171875d043808d0789ac4c374f7d. Latest consensus state: current_term: 1 leader_uuid: "8e29171875d043808d0789ac4c374f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.079260 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:51.079704 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:51.048817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.0.98.61:41047)
0927 15:06:51.048938 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.048942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.061373 (+ 12431us) server_negotiation.cc:407] Connection header received
0927 15:06:51.062188 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.062192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.062253 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.062311 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.064436 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.065203 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.068414 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.068756 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.068833 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:51.072989 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:51.072995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:51.072998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:51.073039 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:51.075026 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.075816 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.075820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.075822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.075920 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.078583 (+ 2663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.078587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.078589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.078831 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:51.078957 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:51.079417 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 15:06:51.079544 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:06:51.082881 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:51.048425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.98.60:39905)
0927 15:06:51.048533 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.048538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.058106 (+ 9568us) server_negotiation.cc:407] Connection header received
0927 15:06:51.063375 (+ 5269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.063380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.063433 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.063488 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.066385 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.067125 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.070756 (+ 3631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:51.071115 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:51.071201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:51.076266 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:51.076271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:51.076274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:51.076303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:51.078085 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.081377 (+ 3292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.081381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.081383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.081479 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:51.081873 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:51.081876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:51.081878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:51.082089 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:51.082259 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:51.082585 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 15:06:51.082723 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:06:51.084887 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e29171875d043808d0789ac4c374f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.085008 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:51.085592 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e29171875d043808d0789ac4c374f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.085721 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:51.086112 22916 mvcc.cc:205] Tried to move back new op lower bound from 7333976518923128832 to 7333976518873640960. Current Snapshot: MvccSnapshot[applied={T|T < 7333976518923128832}]
I20260927 15:06:51.087980 22919 mvcc.cc:205] Tried to move back new op lower bound from 7333976518923128832 to 7333976518873640960. Current Snapshot: MvccSnapshot[applied={T|T < 7333976518923128832 or (T in {7333976518923128832})}]
I20260927 15:06:51.088510 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e29171875d043808d0789ac4c374f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.088635 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:51.089661 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e29171875d043808d0789ac4c374f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e29171875d043808d0789ac4c374f7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } peers { permanent_uuid: "61800bfb9c7a46e1b69ce3ace2b01718" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41047 } } peers { permanent_uuid: "755d9186b2cb40b09f52a013ed2c6bec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39905 } } }
I20260927 15:06:51.089815 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:51.095715 22914 catalog_manager.cc:1347] Generated new cluster ID: 5836c449f4074c12937cd1ca24679938
I20260927 15:06:51.095796 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:51.119752 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:51.120955 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:51.130407 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e29171875d043808d0789ac4c374f7d: Generated new TSK 0
I20260927 15:06:51.131338 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:06:51.154831 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.152911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.98.61:41047)
0927 15:06:51.153164 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.153168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.153606 (+ 438us) server_negotiation.cc:407] Connection header received
0927 15:06:51.153867 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.153871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.153929 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.153995 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.154655 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42696: BlockingRecv error: recv got EOF from 127.0.0.1:42696 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
W20260927 15:06:51.155262 22711 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.152792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.0.98.62:40395)
0927 15:06:51.153052 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.153057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.154269 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 15:06:51.154449 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.154453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.154508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.154562 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.155127 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53422: BlockingRecv error: recv got EOF from 127.0.0.1:53422 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
W20260927 15:06:51.156471 22862 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.153717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.0.98.60:39905)
0927 15:06:51.153885 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.153891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.155674 (+ 1783us) server_negotiation.cc:407] Connection header received
0927 15:06:51.155791 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.155796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.155850 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.155959 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.156352 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56224: BlockingRecv error: recv got EOF from 127.0.0.1:56224 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 15:06:51.159490 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.158373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.98.61:41047)
0927 15:06:51.158480 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.158486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.158652 (+ 166us) server_negotiation.cc:407] Connection header received
0927 15:06:51.158817 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.158824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.158907 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.159000 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.159389 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42704: BlockingRecv error: recv got EOF from 127.0.0.1:42704 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 15:06:51.163412 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.162634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.0.98.61:41047)
0927 15:06:51.162752 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.162755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.162764 (+ 9us) server_negotiation.cc:407] Connection header received
0927 15:06:51.162801 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.162803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.162839 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.162931 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.163317 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42718: BlockingRecv error: recv got EOF from 127.0.0.1:42718 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:06:51.168100 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.167060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.0.98.61:41047)
0927 15:06:51.167161 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.167164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.167361 (+ 197us) server_negotiation.cc:407] Connection header received
0927 15:06:51.167499 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.167502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.167544 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.167635 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.167997 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42730: BlockingRecv error: recv got EOF from 127.0.0.1:42730 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 15:06:51.175210 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.174341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.98.61:41047)
0927 15:06:51.174507 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.174512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.174525 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:06:51.174582 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.174585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.174623 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.174718 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.175116 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42736: BlockingRecv error: recv got EOF from 127.0.0.1:42736 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:06:51.186160 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.185000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.0.98.61:41047)
0927 15:06:51.185115 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.185120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.185320 (+ 200us) server_negotiation.cc:407] Connection header received
0927 15:06:51.185446 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.185451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.185497 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.185584 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.186052 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42752: BlockingRecv error: recv got EOF from 127.0.0.1:42752 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 15:06:51.206027 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.204507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.0.98.61:41047)
0927 15:06:51.204642 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.204648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.205048 (+ 400us) server_negotiation.cc:407] Connection header received
0927 15:06:51.205239 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.205244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.205307 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.205406 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.205901 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42768: BlockingRecv error: recv got EOF from 127.0.0.1:42768 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:06:51.241536 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.240109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.0.98.61:41047)
0927 15:06:51.240257 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.240261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.240598 (+ 337us) server_negotiation.cc:407] Connection header received
0927 15:06:51.240770 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.240773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.240830 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.240920 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.241403 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42778: BlockingRecv error: recv got EOF from 127.0.0.1:42778 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 15:06:51.308630 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.307644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.0.98.61:41047)
0927 15:06:51.307806 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.307810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.307932 (+ 122us) server_negotiation.cc:407] Connection header received
0927 15:06:51.308038 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.308041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.308088 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.308169 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.308536 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42792: BlockingRecv error: recv got EOF from 127.0.0.1:42792 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:06:51.440240 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.438932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.0.98.61:41047)
0927 15:06:51.439071 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.439075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.439208 (+ 133us) server_negotiation.cc:407] Connection header received
0927 15:06:51.439309 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.439312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.439367 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.439514 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.440073 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33360: BlockingRecv error: recv got EOF from 127.0.0.1:33360 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:06:51.699630 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:51.698544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.0.98.61:41047)
0927 15:06:51.698704 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:51.698708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:51.698829 (+ 121us) server_negotiation.cc:407] Connection header received
0927 15:06:51.698895 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:51.698897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:51.698949 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:51.699034 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:51.699475 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33376: BlockingRecv error: recv got EOF from 127.0.0.1:33376 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:06:51.919063 22876 catalog_manager.cc:1259] Loaded cluster ID: 5836c449f4074c12937cd1ca24679938
I20260927 15:06:51.919221 22876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: loading cluster ID for follower catalog manager: success
I20260927 15:06:51.921203 22876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: acquiring CA information for follower catalog manager: success
I20260927 15:06:51.922483 22876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 755d9186b2cb40b09f52a013ed2c6bec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:06:52.031837 22890 catalog_manager.cc:1259] Loaded cluster ID: 5836c449f4074c12937cd1ca24679938
I20260927 15:06:52.031949 22890 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: loading cluster ID for follower catalog manager: success
I20260927 15:06:52.033806 22890 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: acquiring CA information for follower catalog manager: success
I20260927 15:06:52.035094 22890 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61800bfb9c7a46e1b69ce3ace2b01718: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:06:52.218437 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.214127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.98.61:41047)
0927 15:06:52.214435 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:52.214441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:52.214569 (+ 128us) server_negotiation.cc:407] Connection header received
0927 15:06:52.214761 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:52.214764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:52.214934 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:52.215056 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:52.215361 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.215903 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.216819 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.217069 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.217222 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.217528 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:52.218009 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:52.218157 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:52.218206 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 15:06:52.218267 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20260927 15:06:52.225729 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.221799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.0.98.60:39905)
0927 15:06:52.222055 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:52.222059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:52.222074 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:06:52.222269 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:52.222272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:52.222390 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:52.222525 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:52.222774 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.223309 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.224199 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.224424 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.224537 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.224861 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:52.225323 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:52.225465 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:52.225512 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 15:06:52.225568 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260927 15:06:52.233294 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.229171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.0.98.61:41047)
0927 15:06:52.229297 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:52.229304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:52.229659 (+ 355us) server_negotiation.cc:407] Connection header received
0927 15:06:52.229850 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:52.229856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:52.229918 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:52.229984 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:52.230400 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.231148 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.232028 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.232334 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.232430 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.232723 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:52.232826 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:52.232876 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:52.233081 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 15:06:52.233138 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:06:52.233400 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.229054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.0.98.60:39905)
0927 15:06:52.229248 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:52.229254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:52.229634 (+ 380us) server_negotiation.cc:407] Connection header received
0927 15:06:52.229750 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:52.229753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:52.229825 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:52.229891 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:52.230354 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.230883 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.232227 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.232545 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.232655 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.232917 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:52.232999 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:52.233064 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:52.233220 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 15:06:52.233275 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:06:52.234043 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:52.229047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.98.62:40395)
0927 15:06:52.229443 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:52.229447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:52.229463 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:06:52.229565 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:52.229569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:52.229758 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:52.229924 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:52.230150 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.231005 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.232266 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:52.232570 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:52.232694 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:52.233101 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:52.233468 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:52.233559 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:52.233852 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:06:52.233920 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":63,"threads_started":1}
I20260927 15:06:52.237010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22640
I20260927 15:06:52.244711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22715
I20260927 15:06:52.250768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 22789
2026-09-27T15:06:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3258 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:06:52 dist-test-slave-ml7t krb5kdc[23006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:06:52 dist-test-slave-ml7t krb5kdc[23006](info): set up 2 sockets
Sep 27 15:06:52 dist-test-slave-ml7t krb5kdc[23006](info): commencing operation
krb5kdc: starting...
W20260927 15:06:53.314353 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/test-user.keytab.
Sep 27 15:06:53 dist-test-slave-ml7t krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:06:53.439145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35259,127.0.98.61:44497,127.0.98.60:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:06:53.551987 23019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:06:53.552395 23019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:53.552482 23019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:53.552573 23019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:53.556818 23019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:53.556941 23019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:53.556985 23019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:53.557032 23019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:53.557083 23019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:53.562717 23019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35259,127.0.98.61:44497,127.0.98.60:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:53.563910 23019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:53.565012 23019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:06:53.574100 23019 server_base.cc:1034] running on GCE node
W20260927 15:06:53.574039 23025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:53.574120 23024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:53.574023 23027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:53.575006 23019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:53.576335 23019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:53.577546 23019 hybrid_clock.cc:648] HybridClock initialized: now 1790521613577530 us; error 48 us; skew 500 ppm
Sep 27 15:06:53 dist-test-slave-ml7t krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:53.580322 23019 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:06:53.581254 23019 webserver.cc:460] Webserver started at http://127.0.98.62:37413/ using document root <none> and password file <none>
I20260927 15:06:53.581904 23019 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:53.582001 23019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:53.582228 23019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:53.584273 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data/instance:
uuid: "4512bb99552040288bf7c035fbf40bb2"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-ml7t"
server_key: "9967f15a21039a882729ea85642e294e"
server_key_iv: "a17fa0392472b65b5d7cdf01f4cc8193"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.584872 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/wal/instance:
uuid: "4512bb99552040288bf7c035fbf40bb2"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-ml7t"
server_key: "9967f15a21039a882729ea85642e294e"
server_key_iv: "a17fa0392472b65b5d7cdf01f4cc8193"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.588642 23019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:06:53.591295 23034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:53.592551 23019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:53.592696 23019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/wal
uuid: "4512bb99552040288bf7c035fbf40bb2"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-ml7t"
server_key: "9967f15a21039a882729ea85642e294e"
server_key_iv: "a17fa0392472b65b5d7cdf01f4cc8193"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.592816 23019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:53.604140 23019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:53.607534 23019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:53.607748 23019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:53.615370 23019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35259
I20260927 15:06:53.615655 23086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35259 every 8 connection(s)
I20260927 15:06:53.616488 23019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:06:53.619328 23087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:53.623143 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:53.625703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 23019
I20260927 15:06:53.625847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-0/wal/instance
I20260927 15:06:53.626140 392 external_mini_cluster.cc:1428] Setting key b34ddb700b29b0a20d03c0af4e040364
Sep 27 15:06:53 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 15:06:53.640563 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.625432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.0.98.62:35259)
0927 15:06:53.626126 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.626138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.626259 (+ 121us) server_negotiation.cc:407] Connection header received
0927 15:06:53.627181 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.627191 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.627453 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.627667 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.628835 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.629766 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.631617 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.632039 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.632170 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.635085 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.635092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.635113 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.635136 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.637026 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.637531 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.637541 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.637545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.637664 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.638228 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.638231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.638233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.638833 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.638982 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.639406 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.639783 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":580,"spinlock_wait_cycles":45312,"thread_start_us":169,"threads_started":1}
I20260927 15:06:53.640571 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.625424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.0.1:60312)
0927 15:06:53.625949 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.626002 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:53.626853 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:53.627730 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:53.627738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:53.628241 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:53.628633 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.628658 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.629982 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.629989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.631446 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.631462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.632227 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.632232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.632387 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.633055 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:53.633080 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:53.634919 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:53.637154 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.637163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.637174 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.637404 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.637942 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.637948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.637953 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.638114 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.638957 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:53.638963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:53.639092 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 15:06:53.639269 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 15:06:53.639642 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":171,"threads_started":1}
I20260927 15:06:53.643025 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:53.643909 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.643570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53374)
0927 15:06:53.643726 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.643820 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:06:53.644116 23035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
I20260927 15:06:53.645042 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.644721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53376)
0927 15:06:53.644884 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.644960 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:06:53.645216 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44497: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 15:06:53.656618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44497
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35259,127.0.98.61:44497,127.0.98.60:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:06:53.694519 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } attempt: 1
I20260927 15:06:53.695550 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.695165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53390)
0927 15:06:53.695360 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.695447 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 15:06:53.696532 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.696233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53398)
0927 15:06:53.696385 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.696450 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:06:53.696718 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44497: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
I20260927 15:06:53.764926 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } attempt: 2
I20260927 15:06:53.765923 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.765507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53412)
0927 15:06:53.765716 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.765826 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 15:06:53.766743 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.766471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53422)
0927 15:06:53.766608 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.766667 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:06:53.766937 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44497: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44497: connect: Connection refused (error 111)
W20260927 15:06:53.773406 23094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:06:53.773743 23094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:53.773838 23094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:53.773955 23094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:53.778060 23094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:53.778151 23094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:53.778191 23094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:53.778252 23094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:53.778294 23094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:53.783885 23094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35259,127.0.98.61:44497,127.0.98.60:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:53.785051 23094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:53.786217 23094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:06:53.795022 23100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:53.795033 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:53.795081 23099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:53.795282 23094 server_base.cc:1034] running on GCE node
I20260927 15:06:53.795960 23094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:53.797290 23094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:53.798470 23094 hybrid_clock.cc:648] HybridClock initialized: now 1790521613798442 us; error 45 us; skew 500 ppm
Sep 27 15:06:53 dist-test-slave-ml7t krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:53.801743 23094 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 15:06:53.802788 23094 webserver.cc:460] Webserver started at http://127.0.98.61:36443/ using document root <none> and password file <none>
I20260927 15:06:53.803372 23094 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:53.803462 23094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:53.803694 23094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:53.805876 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data/instance:
uuid: "ac33858df74a45a19ab8429e06769cf2"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-ml7t"
server_key: "f825c084f90307df650f048335de42ef"
server_key_iv: "cfa37bf2ac7f824baad88accbf812d67"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.806452 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/wal/instance:
uuid: "ac33858df74a45a19ab8429e06769cf2"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-ml7t"
server_key: "f825c084f90307df650f048335de42ef"
server_key_iv: "cfa37bf2ac7f824baad88accbf812d67"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.810381 23094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:06:53.813122 23109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:53.814584 23094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:06:53.814750 23094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/wal
uuid: "ac33858df74a45a19ab8429e06769cf2"
format_stamp: "Formatted at 2026-09-27 15:06:53 on dist-test-slave-ml7t"
server_key: "f825c084f90307df650f048335de42ef"
server_key_iv: "cfa37bf2ac7f824baad88accbf812d67"
server_key_version: "encryptionkey@0"
I20260927 15:06:53.814875 23094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:53.840824 23094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:53.844203 23094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:53.844437 23094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:53.852046 23094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44497
I20260927 15:06:53.852418 23161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44497 every 8 connection(s)
I20260927 15:06:53.853104 23094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/data/info.pb
I20260927 15:06:53.853816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 23094
I20260927 15:06:53.853962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-1/wal/instance
I20260927 15:06:53.854276 392 external_mini_cluster.cc:1428] Setting key d20feaaed3292df54f252ea91ff468c5
I20260927 15:06:53.856416 23162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:53.858134 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 15:06:53.864132 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:06:53 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:06:53.873108 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.858711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53438)
0927 15:06:53.858943 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.858963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:53.859067 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:53.862183 (+ 3116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:53.862186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:53.862346 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:53.862617 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.862626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.864369 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.864376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.865043 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.865054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.865840 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.865843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.865953 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.866494 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:53.866514 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:53.868233 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:53.870650 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.870654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.870656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.870858 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.871199 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.871202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.871204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.871294 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.872051 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:53.872058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:53.872238 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 15:06:53.872378 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 15:06:53.872871 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":35}
I20260927 15:06:53.874119 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.859494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.61:44497)
0927 15:06:53.859961 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.859981 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.860028 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:06:53.861181 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.861209 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.861709 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.862169 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.862764 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.864224 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.865184 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.865676 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.865889 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.868446 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.868451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.868460 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.868486 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.870537 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.870966 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.870972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.870977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.871088 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.871384 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.871389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.871391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.871894 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.872021 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.872552 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.872850 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":158,"threads_started":1}
Sep 27 15:06:53 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:53.876436 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:53.877305 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.876976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39692)
0927 15:06:53.877127 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.877212 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:06:53.878199 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.877901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39706)
0927 15:06:53.878067 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.878129 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 15:06:53.878427 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
I20260927 15:06:53.879006 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.866722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.0.1:60318)
0927 15:06:53.867760 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.867816 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:53.867962 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:53.868306 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:53.868315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:53.868861 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:53.869075 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.869088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.870221 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.870228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.871482 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.871498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.872209 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.872211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.872292 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.872864 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:53.872877 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:53.874705 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:53.876796 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.876808 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.876814 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.877218 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.877495 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.877498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.877500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.877606 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.878149 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:53.878154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:53.878263 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 15:06:53.878573 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 15:06:53.878753 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":149,"threads_started":1}
I20260927 15:06:53.879477 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.866727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.0.98.62:35259)
0927 15:06:53.866866 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.866875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.867917 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 15:06:53.868072 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.868076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.868130 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.868222 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.869230 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.870076 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.871621 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.871985 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.872076 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.874864 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.874868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.874870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.874896 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.876616 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.877310 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.877316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.877321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.877406 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.877721 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.877725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.877727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.878029 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.878131 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.879213 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.879353 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260927 15:06:53.880468 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 15:06:53.884949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
/tmp/dist-test-taskYaV7G8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37199
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35259,127.0.98.61:44497,127.0.98.60:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:06:53 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:06:53.889264 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.881080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.0.98.61:44497)
0927 15:06:53.881193 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:53.881197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:53.881213 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:06:53.881248 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:53.881251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:53.881297 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:53.881390 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:53.881865 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.882505 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.883330 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.883548 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.883704 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.885951 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:53.885958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:53.885964 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:53.885996 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:53.887506 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.887995 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.887997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.887999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.888074 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:53.888371 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:53.888373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:53.888375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:53.888548 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:53.888655 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:53.888986 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 15:06:53.889127 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:06:53.889307 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.880975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53440)
0927 15:06:53.881089 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.881100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:53.881161 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:53.881431 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:53.881435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:53.881589 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:53.881698 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.881704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.882610 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.882612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.883211 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:53.883220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:53.883730 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:53.883734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:53.883810 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:53.884313 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:53.884328 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:53.885816 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:53.887626 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.887632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.887637 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.887895 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.888168 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:53.888174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:53.888179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:53.888277 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:53.888655 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:53.888661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:53.888775 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 15:06:53.888892 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:06:53.889049 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:06:53.890120 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:53.890949 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.890584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39714)
0927 15:06:53.890764 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.890867 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 15:06:53.891609 23112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
I20260927 15:06:53.892717 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.892419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39720)
0927 15:06:53.892564 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.892627 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 15:06:53.892935 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
I20260927 15:06:53.909622 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } attempt: 1
I20260927 15:06:53.910534 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.910228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39724)
0927 15:06:53.910383 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.910457 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:06:53.911362 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.911145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39728)
0927 15:06:53.911263 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.911310 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:06:53.911535 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
I20260927 15:06:53.942232 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } attempt: 1
I20260927 15:06:53.943282 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.942848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39738)
0927 15:06:53.943052 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.943193 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 15:06:53.944141 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.943881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39740)
0927 15:06:53.943996 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.944059 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:06:53.944289 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
I20260927 15:06:53.986712 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } attempt: 2
I20260927 15:06:53.987600 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.987289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39756)
0927 15:06:53.987448 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.987521 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 15:06:53.988332 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:53.988116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39758)
0927 15:06:53.988227 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:53.988279 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:06:53.988466 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
W20260927 15:06:53.998312 23168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:06:53.998607 23168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:06:53.998667 23168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:06:53.998742 23168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:06:54.002784 23168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:06:54.002856 23168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:06:54.002902 23168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:06:54.002956 23168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:06:54.002993 23168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:06:54.008220 23168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35259,127.0.98.61:44497,127.0.98.60:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:06:54.009313 23168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:06:54.010414 23168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:06:54.012475 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } attempt: 2
I20260927 15:06:54.013322 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.013014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39772)
0927 15:06:54.013158 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:54.013251 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 15:06:54.014106 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.013892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39786)
0927 15:06:54.013998 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:54.014052 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:06:54.014233 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37199: connect: Connection refused (error 111)
W20260927 15:06:54.019131 23175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:54.019114 23177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:06:54.019131 23174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:06:54.019459 23168 server_base.cc:1034] running on GCE node
I20260927 15:06:54.019969 23168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:06:54.021039 23168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:06:54.022233 23168 hybrid_clock.cc:648] HybridClock initialized: now 1790521614022184 us; error 23 us; skew 500 ppm
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:06:54.024883 23168 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 15:06:54.025836 23168 webserver.cc:460] Webserver started at http://127.0.98.60:42097/ using document root <none> and password file <none>
I20260927 15:06:54.026413 23168 fs_manager.cc:359] Metadata directory not provided
I20260927 15:06:54.026504 23168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:06:54.026731 23168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:06:54.028667 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data/instance:
uuid: "3fa916ba1c164122b252717ed6e20794"
format_stamp: "Formatted at 2026-09-27 15:06:54 on dist-test-slave-ml7t"
server_key: "8dbb94707451ebe835ae42537ff2a335"
server_key_iv: "ee9a36450f7fe1a99de3341a337ec4f5"
server_key_version: "encryptionkey@0"
I20260927 15:06:54.029228 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/wal/instance:
uuid: "3fa916ba1c164122b252717ed6e20794"
format_stamp: "Formatted at 2026-09-27 15:06:54 on dist-test-slave-ml7t"
server_key: "8dbb94707451ebe835ae42537ff2a335"
server_key_iv: "ee9a36450f7fe1a99de3341a337ec4f5"
server_key_version: "encryptionkey@0"
I20260927 15:06:54.032881 23168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:06:54.035313 23184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:06:54.036420 23168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:06:54.036556 23168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/wal
uuid: "3fa916ba1c164122b252717ed6e20794"
format_stamp: "Formatted at 2026-09-27 15:06:54 on dist-test-slave-ml7t"
server_key: "8dbb94707451ebe835ae42537ff2a335"
server_key_iv: "ee9a36450f7fe1a99de3341a337ec4f5"
server_key_version: "encryptionkey@0"
I20260927 15:06:54.036660 23168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:06:54.061491 23168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:06:54.064553 23168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:06:54.064772 23168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:06:54.072005 23168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37199
I20260927 15:06:54.072296 23236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37199 every 8 connection(s)
I20260927 15:06:54.073062 23168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/data/info.pb
I20260927 15:06:54.075927 23237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:06:54.079856 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:54.082319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu as pid 23168
I20260927 15:06:54.082477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYaV7G8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521328452215-392-0/minicluster-data/master-2/wal/instance
I20260927 15:06:54.082792 392 external_mini_cluster.cc:1428] Setting key a791be5a5e7bc1c21f84687955d8891f
I20260927 15:06:54.087626 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } attempt: 3
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:06:54.094651 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.081740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.0.98.62:35259)
0927 15:06:54.081910 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.081914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.082887 (+ 973us) server_negotiation.cc:407] Connection header received
0927 15:06:54.084133 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.084135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.084192 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.084270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.085334 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.085894 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.087000 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.087325 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.087405 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.090343 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.090349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.090355 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.090388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.092256 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.092838 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.092845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.092846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.092951 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.093406 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.093411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.093413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.093629 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.093733 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.094387 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.094496 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:06:54.095763 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.082183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.0.1:60334)
0927 15:06:54.082729 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:54.082774 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:54.083890 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:54.084346 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:54.084357 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:54.084807 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:54.085171 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.085201 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.085997 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.086001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.086877 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.086885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.087461 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.087465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.087632 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.088302 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:54.088318 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:54.090203 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:54.092385 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.092398 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.092410 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.092723 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.093053 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.093056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.093058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.093286 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.093809 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:54.093816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:54.094133 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 15:06:54.094346 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 15:06:54.094649 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":34,"thread_start_us":149,"threads_started":1}
I20260927 15:06:54.095770 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.083492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.0.98.60:37199)
0927 15:06:54.083848 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.083858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.083903 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:06:54.084046 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.084073 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.084346 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.084523 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.085224 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.086297 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.087226 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.087678 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.087795 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.090803 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.090810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.090822 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.090846 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.093131 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.093616 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.093622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.093624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.093717 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.094009 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.094014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.094016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.094345 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.094574 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.095010 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.095183 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":26496,"thread_start_us":130,"threads_started":1}
I20260927 15:06:54.097262 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:54.098592 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.088605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39796)
0927 15:06:54.088807 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:54.088823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:54.088905 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:54.089248 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:54.089251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:54.089441 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:54.089577 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.089585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.090444 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.090450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.091274 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.091290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.091887 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.091891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.091994 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.092551 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:54.092566 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:54.093985 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:54.096867 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.096870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.096872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.097114 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.097496 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.097500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.097502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.097613 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.098087 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:54.098090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:54.098188 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 15:06:54.098295 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:06:54.098429 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260927 15:06:54.099478 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.088683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.0.98.60:37199)
0927 15:06:54.089012 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.089020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.089033 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:06:54.089068 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.089073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.089122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.089213 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.089676 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.090320 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.091426 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.091736 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.091857 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.094094 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.094097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.094100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.094123 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.096749 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.097257 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.097260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.097261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.097374 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.097717 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.097720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.097722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.097953 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.098095 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.099213 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.099319 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:06:54.103042 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2: Bootstrap starting.
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:06:54.105392 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } attempt: 3
I20260927 15:06:54.106150 23087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:54.107100 23087 log.cc:824] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:54.107180 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.098054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.0.98.61:44497)
0927 15:06:54.098171 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.098176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.098202 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:06:54.098246 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.098252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.098307 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.098376 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.098895 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.099639 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.100604 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.100925 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.101042 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.103278 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.103282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.103284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.103318 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.105136 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.105727 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.105731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.105733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.105843 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.106161 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.106165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.106167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.106394 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.106472 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.106901 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.107019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:06:54.107556 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.098252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.98.62:35259)
0927 15:06:54.098353 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.098357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.098433 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:06:54.098542 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.098545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.098581 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.098678 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.099307 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.099946 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.100819 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.101102 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.101213 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.103879 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.103883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.103885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.103909 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.105594 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.106100 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.106104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.106106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.106209 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.106540 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.106543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.106545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.106740 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.106877 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.107277 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.107387 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:06:54.108058 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.097753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44497 (local address 127.0.0.1:53444)
0927 15:06:54.097881 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:54.097896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:54.097974 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:54.098490 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:54.098493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:54.098656 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:54.098761 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.098767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.099779 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.099781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.100476 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.100486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.101038 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.101042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.101142 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.101589 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:54.101600 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:54.103077 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:54.105264 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.105268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.105270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.105597 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.105964 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.105967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.105969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.106056 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.106596 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:54.106601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:54.106716 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 15:06:54.107571 (+ 855us) client_negotiation.cc:240] Negotiation successful
0927 15:06:54.107871 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 15:06:54.109558 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:06:54.110597 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2: No bootstrap required, opened a new log
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:06:54.115036 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.105812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39812)
0927 15:06:54.106062 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:54.106077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:54.106161 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:54.106641 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:54.106644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:54.106817 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:54.106917 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.106923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.108012 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.108015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.108836 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.108845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.109585 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.109589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.109689 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.110231 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:54.110250 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:54.111619 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:54.113437 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.113440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.113442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.113645 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.114003 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.114006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.114008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.114098 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.114535 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:54.114538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:54.114664 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 15:06:54.114745 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:06:54.114871 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:06:54.115288 23087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.115552 23087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4512bb99552040288bf7c035fbf40bb2, State: Initialized, Role: FOLLOWER
I20260927 15:06:54.115985 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.106309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.0.98.60:37199)
0927 15:06:54.106420 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.106426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.106441 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:06:54.106474 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.106479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.106525 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.107020 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.107123 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.107887 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.108971 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.109466 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.109922 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.111755 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.111758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.111761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.111786 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.113310 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.113748 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.113750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.113752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.113890 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.114209 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.114212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.114213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.114419 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.115445 (+ 1026us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.115667 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.115830 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":48}
I20260927 15:06:54.115947 23087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.117669 23087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:54.119511 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.109149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.0.98.61:44497)
0927 15:06:54.109246 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.109250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.109532 (+ 282us) server_negotiation.cc:407] Connection header received
0927 15:06:54.109570 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.109572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.109625 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.109693 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.110620 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.111302 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.112387 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.112692 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.112768 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.115779 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.115783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.115786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.115809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.117381 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.117915 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.117918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.117920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.118000 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.118340 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.118344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.118346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.118566 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.118753 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.119142 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.119314 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 15:06:54 dist-test-slave-ml7t krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:06:54.119964 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: Bootstrap starting.
I20260927 15:06:54.123555 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.110542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.0.98.60:37199)
0927 15:06:54.113915 (+ 3373us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.113921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.113936 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:06:54.113972 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.113977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.114073 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.114160 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.114616 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.115344 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.117365 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.117697 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.117815 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.120118 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.120122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.120124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.120150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.121932 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.122408 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.122411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.122413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.122484 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.122719 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.122722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.122724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.122913 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.122980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.123276 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.123397 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3298}
W20260927 15:06:54.123859 23255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:54.123927 23255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:54.123944 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.110366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37199 (local address 127.0.0.1:39826)
0927 15:06:54.110699 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 15:06:54.110718 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:06:54.110797 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:06:54.114260 (+ 3463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:06:54.114263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:06:54.114416 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:06:54.114518 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.114524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.115605 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.115608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.117248 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:06:54.117258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.117940 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.117946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:06:54.118019 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.118464 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:06:54.118482 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:06:54.119951 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:06:54.122053 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.122059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.122064 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.122310 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.122566 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:06:54.122568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:06:54.122570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:06:54.122637 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:06:54.123029 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:06:54.123032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:06:54.123105 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 15:06:54.123672 (+ 567us) client_negotiation.cc:240] Negotiation successful
0927 15:06:54.123798 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":2}
I20260927 15:06:54.124259 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.124352 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:54.126088 23162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:54.126348 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:06:54.127133 23162 log.cc:824] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:54.123574 23019 master_runner.cc:186] Error getting master registration for 127.0.98.61:44497: OK, instance_id { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" instance_seqno: 1790521613849898 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:06:54.128191 23019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:06:54.128273 23061 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4512bb99552040288bf7c035fbf40bb2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60344
I20260927 15:06:54.128568 23061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:06:54.128744 23061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:54.130280 23061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.131958 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: No bootstrap required, opened a new log
I20260927 15:06:54.132118 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: Bootstrap starting.
I20260927 15:06:54.132457 23061 leader_election.cc:288] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [CANDIDATE]: Term 1 election: Requested vote from peers ac33858df74a45a19ab8429e06769cf2 (127.0.98.61:44497), 3fa916ba1c164122b252717ed6e20794 (127.0.98.60:37199)
I20260927 15:06:54.134840 23237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: Neither blocks nor log segments found. Creating new log.
I20260927 15:06:54.135824 23237 log.cc:824] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: Log is configured to *not* fsync() on all Append() calls
I20260927 15:06:54.136678 23162 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.136981 23162 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac33858df74a45a19ab8429e06769cf2, State: Initialized, Role: FOLLOWER
I20260927 15:06:54.137394 23162 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.138801 23162 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:54.139263 23136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4512bb99552040288bf7c035fbf40bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac33858df74a45a19ab8429e06769cf2"
I20260927 15:06:54.139518 23136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:54.141052 23136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4512bb99552040288bf7c035fbf40bb2 in term 1.
I20260927 15:06:54.141573 23035 leader_election.cc:302] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4512bb99552040288bf7c035fbf40bb2, ac33858df74a45a19ab8429e06769cf2; no voters:
I20260927 15:06:54.141916 23244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:06:54.142161 23244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [term 1 LEADER]: Becoming Leader. State: Replica: 4512bb99552040288bf7c035fbf40bb2, State: Running, Role: LEADER
I20260927 15:06:54.142539 23244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.145565 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4512bb99552040288bf7c035fbf40bb2. Latest consensus state: current_term: 1 leader_uuid: "4512bb99552040288bf7c035fbf40bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.146051 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:54.148236 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.135096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.0.98.60:37199)
0927 15:06:54.135252 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.135256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.136954 (+ 1698us) server_negotiation.cc:407] Connection header received
0927 15:06:54.137167 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.137170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.137219 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.137272 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.138356 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.139094 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.140516 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.140889 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.140970 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.142956 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.142960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.142964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.142990 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.144713 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.145197 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.145201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.145203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.145307 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.145620 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.145625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.145628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.145919 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.146055 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.147938 (+ 1883us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.148062 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:06:54.156003 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.137159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.0.98.61:44497)
0927 15:06:54.137266 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.137270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.146787 (+ 9517us) server_negotiation.cc:407] Connection header received
0927 15:06:54.146947 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.146950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.146997 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.147048 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.148411 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.149171 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.150270 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.150597 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.150669 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.152319 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.152323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.152326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.152352 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.154046 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.154519 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.154523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.154525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.154604 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.154870 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.154876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.154879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.155082 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.155227 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.155738 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.155850 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:06:54.158931 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: No bootstrap required, opened a new log
I20260927 15:06:54.159158 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:54.136419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.0.98.62:35259)
0927 15:06:54.136531 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.136536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.141138 (+ 4602us) server_negotiation.cc:407] Connection header received
0927 15:06:54.141335 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.141339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.141400 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.141454 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.142765 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.143763 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.144962 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:54.145328 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:54.145422 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:54.148876 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:06:54.148880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:06:54.148882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:06:54.148907 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:06:54.150672 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.156866 (+ 6194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.156871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.156873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.156971 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:06:54.157736 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:06:54.157740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:06:54.157742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:06:54.158029 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:06:54.158163 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:54.158833 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 15:06:54.158964 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:06:54.154322 23094 master_runner.cc:186] Error getting master registration for 127.0.98.60:37199: OK, instance_id { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" instance_seqno: 1790521614069947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:06:54.161481 23094 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:06:54.161567 23279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:06:54.163034 23237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.163293 23237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa916ba1c164122b252717ed6e20794, State: Initialized, Role: FOLLOWER
I20260927 15:06:54.163621 23237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } }
I20260927 15:06:54.164880 23279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:06:54.165022 23237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:06:54.165100 23211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4512bb99552040288bf7c035fbf40bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3fa916ba1c164122b252717ed6e20794"
I20260927 15:06:54.165320 23211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:06:54.165925 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.166042 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:54.166826 23211 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4512bb99552040288bf7c035fbf40bb2 in term 1.
W20260927 15:06:54.170868 23294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:54.170954 23294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 15:06:54.171263 23280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:06:54.171351 23280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:06:54.171732 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.171883 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:54.181560 23136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [term 1 FOLLOWER]: Refusing update from remote peer 4512bb99552040288bf7c035fbf40bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:54.181560 23211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [term 1 FOLLOWER]: Refusing update from remote peer 4512bb99552040288bf7c035fbf40bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:06:54.182226 23244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:54.182426 23277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:06:54.186126 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4512bb99552040288bf7c035fbf40bb2. Latest consensus state: current_term: 1 leader_uuid: "4512bb99552040288bf7c035fbf40bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.186261 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:54.187623 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4512bb99552040288bf7c035fbf40bb2. Latest consensus state: current_term: 1 leader_uuid: "4512bb99552040288bf7c035fbf40bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.187754 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:54.190451 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4512bb99552040288bf7c035fbf40bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.190577 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:54.190829 23287 mvcc.cc:205] Tried to move back new op lower bound from 7333976531679457280 to 7333976531531534336. Current Snapshot: MvccSnapshot[applied={T|T < 7333976531679457280}]
I20260927 15:06:54.192191 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4512bb99552040288bf7c035fbf40bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.192284 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:54.193956 23296 mvcc.cc:205] Tried to move back new op lower bound from 7333976531679457280 to 7333976531531534336. Current Snapshot: MvccSnapshot[applied={T|T < 7333976531679457280}]
I20260927 15:06:54.197441 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4512bb99552040288bf7c035fbf40bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.197573 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2 [sys.catalog]: This master's current role is: LEADER
I20260927 15:06:54.198583 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4512bb99552040288bf7c035fbf40bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4512bb99552040288bf7c035fbf40bb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } peers { permanent_uuid: "ac33858df74a45a19ab8429e06769cf2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44497 } } peers { permanent_uuid: "3fa916ba1c164122b252717ed6e20794" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37199 } } }
I20260927 15:06:54.198710 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:06:54.199213 23298 mvcc.cc:205] Tried to move back new op lower bound from 7333976531679457280 to 7333976531531534336. Current Snapshot: MvccSnapshot[applied={T|T < 7333976531679457280}]
I20260927 15:06:54.200271 23279 catalog_manager.cc:1347] Generated new cluster ID: 2f0f9132b522404992400da7276b3093
I20260927 15:06:54.200333 23279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:06:54.261366 23279 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:06:54.263023 23279 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:06:54.281101 23279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4512bb99552040288bf7c035fbf40bb2: Generated new TSK 0
I20260927 15:06:54.282112 23279 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:06:54.315269 23241 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.312642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.0.98.60:37199)
0927 15:06:54.312786 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.312791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.314136 (+ 1345us) server_negotiation.cc:407] Connection header received
0927 15:06:54.314284 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.314288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.314344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.314443 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.315114 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39832: BlockingRecv error: recv got EOF from 127.0.0.1:39832 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 15:06:54.315675 23090 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.312141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.0.98.62:35259)
0927 15:06:54.312272 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.312276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.314463 (+ 2187us) server_negotiation.cc:407] Connection header received
0927 15:06:54.314866 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.314870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.314923 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.314975 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.315542 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60360: BlockingRecv error: recv got EOF from 127.0.0.1:60360 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:06:54.318110 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.313603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.0.98.61:44497)
0927 15:06:54.313734 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.313738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.314779 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 15:06:54.317088 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.317091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.317143 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.317200 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.318001 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53466: BlockingRecv error: recv got EOF from 127.0.0.1:53466 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 15:06:54.322618 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.321021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.98.61:44497)
0927 15:06:54.321136 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.321140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.321668 (+ 528us) server_negotiation.cc:407] Connection header received
0927 15:06:54.321904 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.321907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.321974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.322024 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.322512 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53472: BlockingRecv error: recv got EOF from 127.0.0.1:53472 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 15:06:54.327625 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.326237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.0.98.61:44497)
0927 15:06:54.326326 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.326330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.326777 (+ 447us) server_negotiation.cc:407] Connection header received
0927 15:06:54.326962 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.326965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.327015 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.327074 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.327532 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53478: BlockingRecv error: recv got EOF from 127.0.0.1:53478 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 15:06:54.333189 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.331760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.0.98.61:44497)
0927 15:06:54.331870 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.331874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.332304 (+ 430us) server_negotiation.cc:407] Connection header received
0927 15:06:54.332482 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.332485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.332533 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.332582 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.333074 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53492: BlockingRecv error: recv got EOF from 127.0.0.1:53492 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 15:06:54.341132 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.339414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.0.98.61:44497)
0927 15:06:54.339560 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.339564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.340107 (+ 543us) server_negotiation.cc:407] Connection header received
0927 15:06:54.340306 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.340310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.340364 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.340418 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.341010 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53502: BlockingRecv error: recv got EOF from 127.0.0.1:53502 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 15:06:54.353322 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.351594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.0.98.61:44497)
0927 15:06:54.351721 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.351726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.352279 (+ 553us) server_negotiation.cc:407] Connection header received
0927 15:06:54.352498 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.352502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.352561 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.352621 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.353207 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53510: BlockingRecv error: recv got EOF from 127.0.0.1:53510 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 15:06:54.373394 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.371993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53522 (local address 127.0.98.61:44497)
0927 15:06:54.372111 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.372116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.372398 (+ 282us) server_negotiation.cc:407] Connection header received
0927 15:06:54.372535 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.372538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.372627 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.372754 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.373256 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53522: BlockingRecv error: recv got EOF from 127.0.0.1:53522 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 15:06:54.408682 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.407393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.0.98.61:44497)
0927 15:06:54.407563 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.407567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.407686 (+ 119us) server_negotiation.cc:407] Connection header received
0927 15:06:54.407814 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.407817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.407866 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.407957 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.408497 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53532: BlockingRecv error: recv got EOF from 127.0.0.1:53532 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 15:06:54.475943 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.474651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.0.98.61:44497)
0927 15:06:54.474891 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.474896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.474919 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:06:54.475127 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.475132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.475198 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.475290 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.475792 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53534: BlockingRecv error: recv got EOF from 127.0.0.1:53534 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 15:06:54.607300 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.605974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.98.61:44497)
0927 15:06:54.606183 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.606188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.606315 (+ 127us) server_negotiation.cc:407] Connection header received
0927 15:06:54.606495 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.606499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.606556 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.606648 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.607162 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53548: BlockingRecv error: recv got EOF from 127.0.0.1:53548 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:06:54.866783 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:06:54.865514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.0.98.61:44497)
0927 15:06:54.865694 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:54.865698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:54.865881 (+ 183us) server_negotiation.cc:407] Connection header received
0927 15:06:54.866005 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:54.866008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:54.866052 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:54.866126 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:06:54.866672 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53564: BlockingRecv error: recv got EOF from 127.0.0.1:53564 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:06:55.172348 23294 catalog_manager.cc:1259] Loaded cluster ID: 2f0f9132b522404992400da7276b3093
I20260927 15:06:55.172451 23294 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: loading cluster ID for follower catalog manager: success
I20260927 15:06:55.172816 23280 catalog_manager.cc:1259] Loaded cluster ID: 2f0f9132b522404992400da7276b3093
I20260927 15:06:55.172874 23280 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: loading cluster ID for follower catalog manager: success
I20260927 15:06:55.174321 23294 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: acquiring CA information for follower catalog manager: success
I20260927 15:06:55.174525 23280 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: acquiring CA information for follower catalog manager: success
I20260927 15:06:55.175491 23294 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fa916ba1c164122b252717ed6e20794: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:06:55.175556 23280 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac33858df74a45a19ab8429e06769cf2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:06:55.385317 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:55.381137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.98.61:44497)
0927 15:06:55.381408 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:55.381412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:55.381571 (+ 159us) server_negotiation.cc:407] Connection header received
0927 15:06:55.381804 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:55.381808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:55.381949 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:55.382074 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:55.382342 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.382889 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.383792 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.384020 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.384132 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:55.384456 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:55.384895 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:55.385042 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:55.385085 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 15:06:55.385141 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260927 15:06:55.392091 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:55.388400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.0.98.60:37199)
0927 15:06:55.388660 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:55.388665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:55.388689 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:06:55.388762 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:55.388764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:55.388874 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:55.388997 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:55.389218 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.389729 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.390611 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.390844 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.390981 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:55.391283 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:55.391689 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:55.391841 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:55.391879 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 15:06:55.391932 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260927 15:06:55.399968 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:55.395347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.0.98.60:37199)
0927 15:06:55.395697 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:55.395701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:55.395716 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:06:55.395830 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:55.395842 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:55.395909 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:55.395977 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:55.396538 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.397368 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.398522 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.398813 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.398936 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:55.399439 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:55.399512 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:55.399599 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:55.399740 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:06:55.399793 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260927 15:06:55.399979 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:55.395831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.0.98.61:44497)
0927 15:06:55.395965 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:55.395970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:55.395993 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:06:55.396043 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:55.396049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:55.396109 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:55.396183 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:55.396581 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.397184 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.398431 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.398798 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.398915 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:55.399224 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:55.399337 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:55.399423 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:55.399742 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:06:55.399806 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:06:55.400265 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:06:55.395169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.98.62:35259)
0927 15:06:55.395547 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 15:06:55.395552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:06:55.395598 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:06:55.395795 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:06:55.395799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:06:55.395983 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:06:55.396116 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:06:55.396545 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.397441 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.398640 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:06:55.399015 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:06:55.399111 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:06:55.399460 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:06:55.399800 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:06:55.399896 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:06:55.400052 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 15:06:55.400122 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":106,"threads_started":1}
I20260927 15:06:55.408250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 23019
I20260927 15:06:55.414664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 23094
I20260927 15:06:55.420651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYaV7G8/build/debug/bin/kudu with pid 23168
2026-09-27T15:06:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3169 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6428 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286959 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST